DevOps Strategy for Python Functions

The current pipeline: https://github.com/Azure/azure-functions-devops-build/blob/master/sample_yaml_builds/linux/consumption/python-consumption.yml does not work.
Currently the pipeline that uses a bash script is the one that works. This performs CI and CD in one single script, which isn't ideal. There are multiple issues:

Declaring service principal as variables.
No clear separation of CI and CD.

Ideally, the customer expects:

Efficient packing and running from a zip (run from zip on linux will greatly help here)
Deal with virtual environment creation and install of environment dependencies

I know the product is in preview and DevOps task support is early, but this is a placeholder for the same.
Another customer deployment used this pipeline, but customer complains it is very slow, but it is the right way to separate CI and CD.

Thanks for pointing this out @priyaananthasankar. There are some limitations in the approach right now that we're working to address. \cc @Hazhzeng, @Francisco-Gamino, @asavaritayal.

@priyaponders have you tried the azure functionapp devops-build create command to setup the pipeline? This should provision E2E CD without the bash script.

@priyaponders have you tried the azure functionapp devops-build create command to setup the pipeline? This should provision E2E CD without the bash script.

To expand on this, when attempting to setup ci/cd we identified that without zipdeploy being an option, deploying from a server OS seems impossible. Attempting to run "func azure functionapp publish" when using Docker EE on windows server 2019 (which is using LCOW) we get the error "filesystem operations against a running hyper-v container are not supported "

@genscape-agodfrey Sorry for the inconvenience. There's an alternative way to deploy your local function app into Azure.
You may want to try az functionapp devops-build create command in Azure CLI to setup an Azure DevOps build/release pipeline for your functionapp. https://docs.microsoft.com/en-us/cli/azure/functionapp/devops-build?view=azure-cli-latest
Azure CLI is an Azure resource management tool. You may want to download it here.
https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest
If you overcome any issues, feel free to raise them under https://github.com/Azure/azure-functions-devops-build/issues

While not one of the above supported methods, I wanted to use slots for my deployment.  I went with building docker containers & using the az cli to deploy.
My Dockerfile:
FROM mcr.microsoft.com/azure-functions/python:2.0
COPY . /home/site/wwwroot
RUN cd /home/site/wwwroot && pip install -r requirements.txt

In my CD, I use this to set a specific functions container instance to use container version I want:
az webapp config container set --ids ${RESOURCE_ID} --slot ${TARGET_SLOT_NAME} -i ${TARGET_IMAGE_ID}
Later on in the CD pipeline, once I have verified the setup is how I want, I use the following to handle the swapping:
az webapp deployment slot swap --slot ${TARGET_SLOT_NAME} --ids ${RESOURCE_ID}

Closing the issue as there don't seem to be any pending items here.

@asavaritayal a pending item would be allowing the CLI to be used by CI/CD systems besides azure devops. The current approach violates build once, deploy to many.

@genscape-agodfrey  We do not have any plan to add support for 3rd party CI/CD systems to our Azure CLI, other than the current support for Azure Pipelines (DevOps)

@ahmedelnably I would like to build once, deploy many with Azure Pipelines, but the current provided mechanisms do not allow that.
If you note in my comment, I have to package the application by hand and deploy it with az-cli, rather than use the provided tools.

Understood, but that would be a general question about Azure DevOps. creating the deployment artifact can be done either using the Azure Function Core Tools, or from you can also use the DevOps portal and use the built in templates to do the build once,
For the deploy many, you can either use az functionapp deployment config-zip (while using the same zip file for more than one app), or create multiple release pipelines for the deploy many part.
We are also going to get slots support for az functionapp commands in Az CLI 2.0.67 (should be released tomorrow June 18)
More info about these templates are here: https://aka.ms/functions-azure-devops

@ahamad-MS if zip deploys can be made to work for the python functions, then yes, we would be happy using the azure function core tools. Last I checked zip deploys with azure python functions were not possible.


Dependency on com.github.jitpack:gradle-simple:1.0

Hi there,
I am using Maven and I noticed that the talos dependency itself
<dependency>
     <groupId>com.talosvfx</groupId>
      <artifactId>talos-libgdx</artifactId>
      <version>1.4.0</version>
</dependency>

has a dependency on com.github.jitpack:gradle-simple:1.0.

To me that seems strange as it seems to be an example gradle project.
I have not looked at the source code yet, but can someone comment on whether this dependency is in fact needed?
Thanks!

Yep it shouldn't be included.

Closed via #80


split reads including barcode rads while use -br option

I am trying to demultiplex using barcoded reads with -br.
flexbar -r 2_HMWWKBGXB.2.1.fastq.gz -p 2_HMWWKBGXB.2.2.fastq.gz -b barcodes.fasta -br 2_HMWWKBGXB.2.barcode_1.fastq.gz --zip-output GZ
Current output only providing the read1 and read2 only not the barcode one. This would be a great for the single cell analysis especially to the 10x data format
thanks,
Shams

Thanks for your feature request. There is no option for this because the output file names of reads include the barcode name anyway - which can also be the barcode sequence. So I did not see a need for that. But that is not sufficient for your use case, right?
In that case I would consider to include an option for this in a future release.
Kind regards,
Johannes

@jtroehr Thanks for your reply. The need for the barcode is for 10x based single cell analysis using cellranger . It would be really great to include this in the future release.
Thanks,
Shams

Ok, I will let you know as soon as this feature is available
Johannes


Support delayed job execution.

If the delayed job functionality is not configurable (basically we rely on the event having a delay_until field in a very specific format, then delayed job execution becomes trivial the idea is that we dequeue a message, if the message is ready to be executed - we execute. If not - we wait for 10 seconds and put the message back to the queue to give ourselves a chance to process some more messages.
I'm not quite sure how to create a unit test for this though, but manual testing shows it works great. Also perhaps we need to add some instrumentation and custom metrics to the delayed functionality. And it's interesting whether you think relying on the delay_until field format and not making this configurable is ok.
Bug: T186261
cc @wikimedia/services

Another question here is that putting the message back to the same topic will not play well with the metrics that we use - we will basically report the same message over and over again and we will not be able to see the actual job enqueue rate. We could avoid that by using a special new .delay topic and subscribing to it by default, but technically that's not required and also will tie up a precious thread from the libuv thread pool.

Another question is whether we want to try deduplication upon intermediate dequeing - there's no need to put the event back into the queue if we know that it will get deduplicated on execution - however, this requires some custom deduplication logic just for the delayed jobs and it's not very easy to make - I propose to treat this kind of deduplication as an optimization and implement it later if we find out that trafficfor delayed job topics is a problem.

Coverage decreased (-0.4%) to 89.281% when pulling b8b019b0a043cf14cee8f1b1600299833b288db0 on Pchelolo:delayed_execution into 33e0e5b58e82699d8d06ee174667c747c11cfe3b on wikimedia:master.


Add missing archivers to workflow. Also the production FERC1 DOI.

I also alphabetized the datasets in the workflow.

Consistently having trouble with the Census DP1 archive. Before it was pointing at an archive from 2020 as the original, and I updated it to point at a new "original" that was created in January instead (probably with the current archiving infrastructure) and it still fails, but gets to the point of trying to make a new version now.
Seems like it's the Zenodo connection that's failing?
2023-02-21 21:10:09 [    INFO] catalystcoop.pudl_archiver.archivers.classes:71 Archiving censusdp1tract
2023-02-21 21:10:09 [    INFO] catalystcoop.pudl_archiver.depositors.zenodo:91 GET https://sandbox.zenodo.org/api/deposit/depositions - Query depositions for 10.5072/zenodo.1151526
2023-02-21 21:10:10 [    INFO] catalystcoop.pudl_archiver.depositors.zenodo:91 GET https://sandbox.zenodo.org/api/deposit/depositions/1151527 - Get deposition for 1151527
2023-02-21 21:10:11 [    INFO] catalystcoop.pudl_archiver.depositors.zenodo:91 POST https://sandbox.zenodo.org/api/deposit/depositions/1151527/actions/newversion - Creating new version
2023-02-21 21:10:13 [    INFO] catalystcoop.pudl_archiver.depositors.zenodo:91 PUT https://sandbox.zenodo.org/api/deposit/depositions/1166412 - Updating version number from 1.0.0 (1151527) to 2.0.0
2023-02-21 21:15:14 [    INFO] catalystcoop.pudl_archiver.utils:46 Error while executing <coroutine object ClientResponse.read at 0x7f00a381cf90> (try #1, retry in 2s):
2023-02-21 21:15:16 [    INFO] catalystcoop.pudl_archiver.utils:46 Error while executing <coroutine object ClientResponse.read at 0x7f00a381d150> (try #2, retry in 4s): Connection closed
2023-02-21 21:15:20 [    INFO] catalystcoop.pudl_archiver.utils:46 Error while executing <coroutine object ClientResponse.read at 0x7f00a381cf90> (try #3, retry in 8s): Connection closed
2023-02-21 21:15:28 [    INFO] catalystcoop.pudl_archiver.utils:46 Error while executing <coroutine object ClientResponse.read at 0x7f00a381d150> (try #4, retry in 16s): Connection closed
2023-02-21 21:15:44 [    INFO] catalystcoop.pudl_archiver.utils:46 Error while executing <coroutine object ClientResponse.read at 0x7f00a381cf90> (try #5, retry in 32s): Connection closed
2023-02-21 21:16:16 [    INFO] catalystcoop.pudl_archiver.utils:46 Error while executing <coroutine object ClientResponse.read at 0x7f00a381d150> (try #6, retry in 64s): Connection closed

I logged in and found that there were draft depositions for the Census DP1 for both the old and new archive lineages -- it had uploaded the files but they didn't get published.


test immutableadts

no tests were written for this collection, would be nice to have some tests

tests added in: d440fe27849467a601a827243d1af761c76cd157


C++/WinRT should assert that projected types are always the same size as ABI types

From internal bug report:
winrt::array_view is a bit too generous?
struct awesomeString : hstring { int extra; };

std::vector<awesomeString> a = ...;

imagePicker.FileTypeFilter().ReplaceAll(a); // This compiles!

I passed a vector of awesomeString objects, which are not ABI-compatible with hstrings, and nobody complained. Because awesomeString is implicitly convertible to hstring.
This seems to be intentional across the array_view constructors.
template <typename C>
array_view(std::vector<C>& value) noexcept :
    array_view(value.data(), static_cast<size_type>(value.size()))
{}

We let you pass a vector / conformant array / std::array of C (not just T) – as long as C* is convertible to T* we decide that it’s okay. But there’s no check that C is ABI-compatible with T. Should we at least have
static_assert(sizeof(C) == sizeof(T), "Conformant array of incompatible type");

? I’m guessing we’re generalizing to C so that you can do
struct awesomeString : hstring { bool IsAwesome(); };

which adds methods to an existing type without changing its ABI. But we should check that you are in fact not changing the ABI. (But we don’t extend the same courtesy to initializer_list; that one must be exactly initializer_list.)
This constructor can also get faked out:
array_view(pointer first, pointer last) noexcept :
    m_data(first),
    m_size(static_cast<size_type>(last - first))
{}

imagePicker.FileTypeFilter().ReplaceAll(a.begin(), a.end()); // Compiler doesn’t complain!

The compiler is happy with this because a.begin() and a.end() are awesomeString* which is convertible to hstring*. What it doesn’t know is that the iterators are not actually going to work for iterating over hstrings.
Should this be
template<typename Iter>
array_view(Iter first, Iter last) noexcept :
    m_data(first),
    m_size(static_cast<size_type>(last - first))
{
    static_assert(sizeof(*first) == sizeof(T), "Incompatible iterator");
}

Moved to https://github.com/microsoft/cppwinrt/issues/358


onPress mapview doest return coordinate

I want to draw marker when map is pressed. When i log the onPress(console) doesnt give return coordinate but when I use onRegionChange it does return coordinate. I use this lib in android (react-native project).
onRegionChange={(region) => {              console.log(region);            }}
perfectly works
onPress={(coordinate) => {              console.log(coordinate);            }}
doesnt work.
Somebody help me.
Great Project by airbnb!!! thanks

It doesn't work in my app too.

its working by using e.nativeEvent.coordinate
so the log is console.log(e.nativeEvent.coordinate)
@zuber303

onPress={(e.nativeEvent.coordinate) => { console.log(e.nativeEvent.coordinate) }} doesn't work

Thanks @benyliantriana, this is annoying... The way that MapView api is written today leads us to believe that {coordinate, point} params get injected directly to onPress callback but it does not. To make matters worse, when you try to log what params you truly get the app crashes with there was a problem sending log messages to your development environment because nativeEvent object is way too large for log stream, leaving us stranded as to how to get to the params until getting here. Why make the api docs unclear?

to know what's inside received object you can easily console.log() or alert() it.
and because it have cyclic data you can use following library to print it's content
import {decycle} from 'json-cyclic'

function onPress(data) {
    console.log(JSON.stringfy(decycle(data), null, '\t'))
}

onPress={ (event) => console.log(event.nativeEvent.coordinate) } Worked for me

Thanks


asyncio typesignatures

PR 3/5 for asyncio support in mediagrains.
Follows after: #55
Contains type hinting for grain construction
Job: https://www.pivotaltracker.com/story/show/168535056

I considered that, but I think there's enough processing that needs doing on the parameters in place that it's easier to have them all named like this rather than using a dictionary.


ci(action): further reduce load on CI

trigger on 'merge_group' and 'push to main' only if changes in docker files, so that pull requests do not trigger so many jobs
add weekly build (Sunday midnight)

mistake


[Feature Request] External CPI calls should eventually timeout

Is your feature request related to a problem? Please describe.
There's a few reasons CPI calls may hang. We've seen the Director's Open3 call to the CPI binary fail to spin up the CPI but block on the spawned process, or a call between the cpi and IAAS may have network issues. As a result, the bosh task backing the call hangs indefinitely in a "running" state until manually cancelled.
Describe the solution you'd like
There should be a -very long- timeout on bosh's side to cpi calls (hours? we should probably look into what the most expensive possible calls could be). Having tasks fail with an error message prompting users to look at the cpi logs would help.  It's also likely that if it is flake, users can deploy their way forwards with minimal disruption.
Describe alternatives you've considered
Timeout's can be implemented on the cpi side - but there's a fan out issues as there are multiple CPI's. It also doesn't help with possible director -> cpi hanging communication issues.

Here's an instance https://cloudfoundry.slack.com/archives/C02HPPYQ2/p1573162453274000 where it looks to have hung connecting to the IAAS. It does look like the vsphere cpi at least has some timeout functionality

I think that it is better to have CPIs with properly configured IaaS/HTTP clients, because the CPIs can better decide when to timeout. E.g. for the OpenStack CPI connection options are even configurable via the bosh manifest. You can set http (excon) options depending on the OpenStack needs. There are also situation, where the CPIs retry and that is why it takes too long. A general safeguard on the bosh side, which can terminate the operation if something takes too long is a good idea. It is just difficult to find an optimal timeout, but a "very long" timeout is for sure better than no timeout.

Another suggestion that Kevin brought up is having the cpi's log somewhere themselves rather than waiting for bosh to log the cpi's actions after it finishes. It would make debugging what goes wrong easier. Even in the retry issue you linked above, it didn't show logs until after the 5 hours had passed and it gave up and returned which is not an optional operational experience.

Saving the cpi's log somewhere or consuming them differently sounds to me as a prerequisite for this feature, because I'm not sure what happens if bosh has to terminate the cpi process. I think in this case the process logs are gone. But this is a good point with the cpi's log.

To wrap up what I think we've covered, I think this is a more concrete form of what we've discussed above:

Switch the directors reading of the cpi logs from a final (at exit of binary) read of logs to a streaming input - should be a switch from using Open3's capture3 to popen3. This means that we don't need to ask the CPI's to do anything differently too.

Add a 3 hour timeout to the cpi calls. I've made up the number but I can't think of situations in which a user would want longer. Even in the AWS We currently do not have sufficient m4.4xlarge capacity in the Availability Zone... case, I would think it would be more valuable for Operators to have the option of switching AZ or VM type. We could also make this configurable if there's pushback. If you have a better initial number to propose here, I'm happy to use that too.

I think it's also a given that we'll keep trying to push back/fix CPI's that are failing for reasons like taking too long or failing to timeout on calls. Hopefully the better logs will help us and another CPI team's find solutions faster.

Sounds reasonable. I was thinking for a feature flag, but we can wait for a pushback also on this.


extended connector  with subprocesses and functions for downloading f…

Hi reguel,
can you please review my changes for the docuware connecter (+ demo project)
as we discussed a few days ago i added some sub processes and functions mainly for uploading files inculding index fields (meta data) and for updating these index fields
Many Thanks
Ramazan

Hi Reguel,  thank you for the review. i tried to fix the points. Besides that i added BPM Tests for the sub-processes

Hi Reguel, thank you for the review. i tried to fix the points. Besides that i added BPM Tests for the sub-processes

Thanks a lot @ramazanakbulut . For me the this change is good and well covered with tests.

I merged it with :crossed_fingers:  can you verify whether this is a true error stated in the pipeline?
or a false positive?

ah sorry, i forgot to update this in the demo project. i just removed the wrong return parameter

can you fire a fix PR for this @ramazanakbulut ?
our build monitor is blaming this repo for its quality -> https://axonivy-market.github.io/market-monitor/


Fix: edge cases in std/algebra elliptic curve arithmetic circuit (emulated and 2-chains)

Description

Fix edge cases in std/algebra involving scalars 0 and points (0,0) using the UseSafe option.
Fix bug when scalar is 1 in scalarMulGeneric, scalarBitsMulGeneric and related methods.
Fix #964 (but the fix is commented out to save constraints for the cases with appi.Commit that we use in Linea)

Type of change

[x] Bug fix (non-breaking change which fixes an issue)

How has this been tested?

Current tests
New tests for missing methods
New tests taking into accounts all edge cases

How has this been benchmarked?

Constraints count doesn't change when UseSafe option is not used (except for scalarMulGeneric), which is what interests us the most for our use-cases. The option is to capture all edge cases in all use-cases.
Best thing forward is to add relevant emulated circuits to internal/stats.
Checklist:

[x] I have performed a self-review of my code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] I have added tests that prove my fix is effective or that my feature works
[x] I did not modify files generated from templates
[x] golangci-lint does not output errors locally
[x] New and existing unit tests pass locally with my changes
[x] Any dependent changes have been merged and published in downstream modules

Looks good, however I think for scalarmul by constant we can have fast paths which do not perform computations.
Dunno about PLONK verifier options though - I think they are good to have but not essential. I think we can implement them later (but creating an issue for tracking), it is up to you.

Applied fast path for constScalarMul when s==0. For the case s==1 I didn't do it since we never use this method anyway we can also specialize paths for s==2 and so on.. but s==0 is just to have the exposed method working for all cases even if it's never used in our use-cases. For the plonk option I will create an issue and close this PR. Wdyt?


Container security operator display name changed

The container security operator now has a new display name.
This is causing problems with the CSV check in the automation.
Signed-off-by: Gus Parvin<EMAIL_ADDRESS>

/unhold


silx.io.specfile: Fixed buffer overflow for too long motor or label

This PR makes the temporary buffer larger (256 chars) and cuts the string in case of overflow.
There is certainly a better way to fix this by copying directly the right part of buf into the newly allocated onelabel buffer and avoiding the use of a temporary buffer, but it involves more code changes.
closes #3621

I wonder if SPEC itself allows to have a label of more than 40 characters.

Provided this issue is there since "ever", I would say no, but from a very quick look at the documentation I could not find something about it.


Refactor driveteam controls

Open issue for discussion: what buttons should we have and where?
Driver:

[x] Left/right sticks: tank drive
[x] Left/right trigger: throttles
[x] Left bumper hold: shoot

Copilot:

[ ] Left/right sticks: climber extension
[x] D pad up/down: climber solenoids
[x] A: "Acquire ball" (fold down and spin intake, start indexer)
[x] B: Outtake
[ ] Y: Eject only top ball through shooter
[ ] X: Eject only bottom ball through intake

For the X can't I just outtake?

No, we may want to remove the bottom ball only, not the top ball. Reverse-intaking will get rid of both


Make bugs format like issues count

The bugs count should format like the issues count.

Please follow the issue template

Here

What do you want changed?

The 1,037 should be 1.3K, like how the issues tab is formatted

Indeed this happens as soon as they hit 1000. I think it didn’t before
Example repo with 1000 issues: https://github.com/nodejs/node/issues

Indeed:


write your own transpiler pass

This actually closes #106.

Looks like you still have the stray file defaults_and_configuration_options.mdx. You can reset it to the main branch by doing something like git checkout main docs/transpile/defaults_and_configuration_options.mdx and then committing.

You ended up deleting the file which might cause conflicts later, which is why I suggested simply resetting it with git checkout.

You ended up deleting the file which might cause conflicts later, which is why I suggested simply resetting it with git checkout.

whoops i'll try again

Thanks both! I just pushed some small updates to typos, linting, rephrasing etc. @beckykd could you give it a last copy edit and then should be good to go? In particular not entirely sure I added the images to the folder correctly

Thanks both! I just pushed some small updates to typos, linting, rephrasing etc. @beckykd could you give it a last copy edit and then should be good to go? In particular not entirely sure I added the images to the folder correctly

Done - the images look fine to me.

I just noticed that the title is "Write your own transpiler pass".  I feel like a better title would be "Write a transpiler pass" or "Write a custom transpiler pass".  Thoughts, @javabster @nbronn ?
I know it's been merged, but I could open a new PR to change it after the merge

@beckykd sure feel free to open a followup PR :) I like "Write a custom transpiler pass"

I just noticed that the title is "Write your own transpiler pass". I feel like a better title would be "Write a transpiler pass" or "Write a custom transpiler pass". Thoughts, @javabster @nbronn ?
I know it's been merged, but I could open a new PR to change it after the merge

I think there's a lot of things you can customize, so the word "custom" lost a bit of meaning in these discussions. I kind of like the title now, as "Write a transpiler pass" sounds kind of bland.

I just noticed that the title is "Write your own transpiler pass". I feel like a better title would be "Write a transpiler pass" or "Write a custom transpiler pass". Thoughts, @javabster @nbronn ?
I know it's been merged, but I could open a new PR to change it after the merge

I think there's a lot of things you can customize, so the word "custom" lost a bit of meaning in these discussions. I kind of like the title now, as "Write a transpiler pass" sounds kind of bland.

The problem is that we want to avoid the use of "you" and "we", so I'd rather not put it in the title.  Plus it's just an extra word

Why not just use "Writing transpiler passes"?

Why not just use "Writing transpiler passes"?

Or "Write transpiler passes", because titles are not supposed to start with gerunds ('ing')


Get (real) current cursor position

I have a scenario where I need to insert text at a certain index. In this issue you specified you can use a list of range functions to do that. However, I noticed that those functions (especially getSelection()) doesn't give the proper index.
If I type: Hello World the underlying HTML is equal to:
Hello World
. if I then set my cursor to the index right before the W, getSelection() gives back all kinds of offsets with the number 6. Which would be correct if the text is only Hello World, but it's not, the HTML includes the p tags, which would make the real character position 9.
Is there a way to get the index that includes counting all of the HTML characters?

@guido-visser
If what I understand is getting an index containing the index of the
 HTML string, it's impossible.
Get the index of the text, then use util.getFormatElement(this.getSelectionNode(), null) to get the name of the current tag and calculate it.


Discuss: Incremental linting beta

LAMS v3.1 will introduce the ability to do incremental linting, marked as beta. This issue will provide a space for people to share their feedback.

Hi @jolfad ! Thanks for taking the time to provide some feedback!
Pruning the unused exemptions is a great suggestion, I'll plan on adding that functionality.
As for the source argument, it should be a single string value, which will be passed to the glob npm module. So, like --output="**/{*.model,*.explore,*.view,manifest}.lkml". Perhaps your suggestion would be the more "CLI convention following" way to accept these arguments, which I should also support...

I just tried this out, and I can see that it adds exemptions to the lams-exemtions.ndjson file. However, it doesn't seem like these exemptions are automatically removed from the file once the issues are fixed. This means one needs to manually maintain and remove exemptions. I think this is something that could be nice to include.

Writing just to say I agree with @jolfad that native pruning of unused exemptions would be very helpful for making this work seamlessly as a CI check in Github Actions, for example.


[SPARK-21102][SQL] Make refresh resource command less aggressive in p…

Idea
This PR adds validation to REFRESH sql statements. Currently, users can specify whatever they want as resource path. For example, spark.sql("REFRESH ! $ !") will be executed without any exceptions.
Implementation
I am not sure that my current implementation is the most optimal, so any feedback is appreciated. My first idea was to make the grammar as strict as possible. Unfortunately, there were some problems. I tried the approach below:
SqlBase.g4
...
    | REFRESH TABLE tableIdentifier                                    #refreshTable
    | REFRESH resourcePath                                             #refreshResource
...

resourcePath
    : STRING
    | (IDENTIFIER | number | nonReserved | '/' | '-')+ // other symbols can be added if needed
    ;

It is not flexible enough and requires to explicitly mention all possible symbols. Therefore, I came up with the current approach that is implemented in the code.
Let me know your opinion on which one is better.

Test build #78320 has started for PR 18368 at commit fc2b7c0.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78320/
Test FAILed.

test this please

Test build #78338 has started for PR 18368 at commit fc2b7c0.

@shaneknapp can we trigger this one more time, please?

test this please

Test build #78596 has started for PR 18368 at commit fc2b7c0.

Test build #78596 has finished for PR 18368 at commit fc2b7c0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78596/
Test FAILed.

Merged build finished. Test FAILed.

so yeah...  some of these PRs created during the "troubles" we had over the past couple of weeks aren't responding well to the PRB trigger.  we have a couple of options here:

close this PR and open a new one (on a new branch), and things will just 'work'.
i can babysit this PR if (1) turns out to be too onerous.

let me know how you'd like to proceed...
shane

Jenkins, retest this please.

test this please

Test build #78667 has started for PR 18368 at commit fc2b7c0.

Test build #78667 has finished for PR 18368 at commit fc2b7c0.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78667/
Test FAILed.

@shaneknapp It seems that the build fails with an exception non-related to the PR. Therefore, I will just close this one and open a new one.

@aokolnychyi It won't help; the build is broken right now. Just leave this and retest when it's resolved

retest this please

Test build #78703 has started for PR 18368 at commit fc2b7c0.

Test build #78703 has finished for PR 18368 at commit fc2b7c0.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78703/
Test FAILed.

Merged build finished. Test FAILed.

Test build #79051 has started for PR 18368 at commit 6ab9191.

Test build #79051 has finished for PR 18368 at commit 6ab9191.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79051/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Test build #79053 has started for PR 18368 at commit 6ab9191.

LGTM

Test build #79053 has finished for PR 18368 at commit 6ab9191.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79053/
Test PASSed.

Merged build finished. Test PASSed.

@aokolnychyi Could you please fix the PR title?
[SPARK-21102][SQL] Refresh command is too aggressive in parsing

Thanks!

@gatorsmile should be fixed now.

Thanks! Merging to master.

Hi, @gatorsmile .
Could you set @aokolnychyi as an Assignee and Fix Versions at https://issues.apache.org/jira/browse/SPARK-21102? Thanks!

Set.


Issues signing in and out after upgrade (0.2.12)

Do you want to request a feature or report a bug?
Bug
What is the current behavior?
Sign in and sign out stopped working in 0.2.12. Unsure exactly which version we had installed prior to the upgrade, but our package.json has ^0.2.9 in it.
If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem. Your bug will get fixed much faster if we can run your code and it doesn't have dependencies other than AWS Amplify.
After upgrading with npm upgrade our existing sign in and out calls began to fail (with 400s).
Sign in gave the error signin failed but still logged the user in (upon reloading the currentAuthenticatedUser returned a user), sign out then gave the error Failed to load creadentials, and did not sign the user out, so they were stuck inside our app.
What is the expected behavior?
Calling sign in and sign out should not error.
Which versions of Amplify, and which browser / OS are affected by this issue? Did this work in previous versions?
Browser: Chrome 65.0.3325.181 (Official Build) (64-bit)
OS: Mac OS X El Capitan 10.11.6
Node: 6-8
Used to work in 0.2.9 and possibly up to 0.2.11. Broke at some point between 0.2.9 and 0.2.12.
You can turn on the debug mode to provide more info for us by setting window.LOG_LEVEL = 'DEBUG'; in your app.

@JakeSidSmith this issue seems to be related to #510 which will happen when unauthenticated access is not enabled in the identity pool.

Signing in with debug enabled.

Need any more detailed screenshots?

@JakeSidSmith I had this issue too which is caused by caching and did some fix around that yesterday. Could you try the latest version of amplify to see it works? And also could you check that bad http request to cognito?

Have just tried 0.2.14.
Signout fails, but returns no errors.
Upon reload currentAuthenticatedUser still returns the user.
Signin still gives signin failed error.
Here's the signout call:

Hi @powerful23 I'm working on this with @JakeSidSmith . We're a little confused as to what we need to set up. We are not actually using Identity Pools. Instead we have a User Pool which we are using in API Gateway to authorise against.
From what I can tell, Identity Pools are used to grant federated access to AWS services which isn't what we're doing so granting unauthenticated access also doesn't make too much sense.

@mathewpower we use the jwt token from the user session to create an identity in that identity pool so that the user can get an aws credentials which allows him to use other aws services like S3, Pinpoint. For now we bind these two together(user pool and identity pool) as this is the most common use case but yes, we might need to support the case if developer just want to use the User Pool to get user sign in. We will have a work around this next week.

Closing this issue as it should have been fixed in the latest version.

I'm in same boat. I have Amplify somehwat working without cognito identity pool now, but it still throws errors to me about the identity pool not being set.
'amazon-cognito-identity-js' (which seems to be inappropriately named, as it's for user pools?) has been sucked in to this repo as a child component. That sends a pretty strong signal that anyone who wants to use user pools at all is supposed to use Amplify. But I'm constantly finding that I have to fight around all the other features/federation that is being pushed on me. It's the simplest use case.. surely it's one of the most common?
Should we be using Amplify?

POST https://cognito-identity.us-west-2.amazonaws.com/ 400 () when signing out


Add a list of popular websites which people mostly visit.

The ChromeActivity has  a list of strings  url.
Add popular websites ike  sites like medium ,reddit to it with the help of:

List.add(String)

method.
For e.g.

url.add("https://medium.com");

I would like to work on this! Is there any prefered amount/limit on how many websites to add?

Preferably there is no limit  but addition of uncommon sites should be avoided.

#7 has about 80 links now, hope it helps :)

I have some more that I could add! Is it also chill if I organize the links by categories(i.e news sites, social media, ect.)?


Tendermint doesn't creates block in kv example

What went wrong?
Tendermint doesn't creates block in kv example.

Steps to reproduce
Just start the kv example included in /abci/README.md
Definition of "done"
Tendermint should start to create blocks and start the rpc.

What version of tendermint-rs are you using? And are you using CometBFT or Tendermint Core, and which version?

I am using github as the version(git = "..."), and i am using tendermint core. Which is version 0.34.21

The tendermint-abci crate on main only supports the latest version of CometBFT, which is v0.38 (currently a release candidate).

The tendermint-abci crate on main only supports the latest version of CometBFT, which is v0.38 (currently a release candidate).

Thanks for the reply, it works like a charm. Have a great day!


Change hive.txn.manager to DummyTxnManager

Current hive.txn.manager is org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager.
If Kyuubi's HiveSQLEngine crashed and restarted, following queries may wait for Table lock for a long time until last lock is expired.

cc @pan3793


Documention update around format

Currently the default format is a PSV file that looks similar to the output from bgpdump. Documentation for this output would be great. Similarly for the JSON file, I'm confused how the PSV is converted into the JSON file.
I think as a short term fix you mentioned merely viewing: https://github.com/bgpkit/bgpkit-parser/blob/33618df32f148aad846cd7aef8cc78e2c49eb422/src/models/bgp/elem.rs#L126

The generation of PSV is manual as you linked there. The JSON output is done automatically by the Rust's serde crate. In essence, it takes the content of BgpElem struct and convert it to JSON string automatically.
You will have to look at the BgpElem fields' types for their conversion details:
#[derive(Debug, Clone, PartialEq)]
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
pub struct BgpElem {
    pub timestamp: f64,
    #[cfg_attr(feature = "serde", serde(rename = "type"))]
    pub elem_type: ElemType,
    pub peer_ip: IpAddr,
    pub peer_asn: Asn,
    pub prefix: NetworkPrefix,
    pub next_hop: Option<IpAddr>,
    pub as_path: Option<AsPath>,
    pub origin_asns: Option<Vec<Asn>>,
    pub origin: Option<Origin>,
    pub local_pref: Option<u32>,
    pub med: Option<u32>,
    pub communities: Option<Vec<MetaCommunity>>,
    pub atomic: bool,
    pub aggr_asn: Option<Asn>,
    pub aggr_ip: Option<BgpIdentifier>,
    pub only_to_customer: Option<Asn>,
    /// unknown attributes formatted as (TYPE, RAW_BYTES)
    pub unknown: Option<Vec<AttrRaw>>,
    /// deprecated attributes formatted as (TYPE, RAW_BYTES)
    pub deprecated: Option<Vec<AttrRaw>>,
}

Thank you!