Save cleaned data of Ingestion Server to AWS S3

Fixes

Fixes #3912 by @krysal
Description

This PR changes the Ingestion Server's cleanup step to temporarily save the cleaned data to disk and upload the files to an S3 bucket once they reach a certain size (chose 10 MB kind of arbitrarily, to make them manageable). S3 does not support appending to files; you can only replace them, so the file for each column to modify is split into chunks and uploaded as they are generated.
Looking at this code line in the same Ingestion Server makes me guess the credentials are taken from the environment variables and nothing else is needed in production to connect to S3, but I can't guarantee it. Let me know your thoughts on this approach.
https://github.com/WordPress/openverse/blob/9ac2586cfb3a4c7bb223e232f79506f1ee3c2635/ingestion_server/ingestion_server/distributed_reindex_scheduler.py#L26
Testing Instructions

To test this locally with MinIO, create the openverse-catalog bucket if it's not automatically created at start. Go to http://localhost:5011/ (username: test_user & password: test_secret).
just api/up && just catalog/up

Then make some rows of image table in the catalog dirty by removing the protocol in one or several of the URLs (url, creator_url, or foreign_landing_url) or modify the tags for low accuracy. Run the data refresh from the Airflow UI, wait for it to finish and check the bucket in MinIO.
http://localhost:5011/browser/openverse-catalog/
The data refresh should continue even if the upload to S3 fails for whatever reason. Try shooting down the S3 container and clearing the ingest_upstream step in the DAG to confirms it continues despite the failure in the upload.
Checklist

[ ] My pull request has a descriptive title (not a vague title likeUpdate index.md).
[ ] My pull request targets the default branch of the repository (main) or a parent feature branch.
[ ] My commit messages follow best practices.
[ ] My code follows the established code style of the repository.
[ ] I added or updated tests for the changes I made (if applicable).
[ ] I added or updated documentation (if applicable).
[ ] I tried running the project locally and verified that there are no visible errors.
[ ] I ran the DAG documentation generator (if applicable).

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

When running locally, I got the bucket does not exist error. Do you think it's better to reuse an existing bucket, or to add the new one to the minio docker template:
BUCKETS_TO_CREATE=openverse-storage,openverse-airflow-logs

I checked in S3 and we've used s3://${OPENVERSE_BUCKET}/shared/data-refresh-cleaned-data/ for previous versions of these files, so I think that would be okay to use!

upload the files to an S3 bucket once they reach a certain size (chose 10 MB kind of arbitrarily, to make them manageable)

Having checked S3, it looks like the creator_url.tsv and foreign_landing_url.tsv (the largest ones) are 700-800MB (IIRC tags.tsv was too large to even save and upload). We have 16GB of storage available on the primary ingestion server and 8GB available on each indexer worker. Given that a 10MB limit would produce 80 files for creator_url.tsv for example, I think setting the limit to something like 1GB might make more sense!

@obulat @AetherUnbound Thanks for your valuable suggestions. I was too quick to mark this as ready before but I'm still glad I did!
@AetherUnbound  Regarding the file size, I set it to 850 MB, so only one file is created for each field but tags. 1 GB is on the big side for downloading without a high-speed connection. About the duplicated rows in the file, I'm not sure what could have happened there; I don't see those results in my tests 🤔 Could you try again? I applied the rest of changes suggested.

@AetherUnbound After following those instructions, I also get some duplicates. It's weird, but I think it might be related to the retries for the image data refresh process locally.
https://github.com/WordPress/openverse/blob/feb8e003fe34db0fa744c7120a99abef839bf0e3/load_sample_data.sh#L130-L138
I added a specific folder for the output files, which gets recreated every time the process begins. I don't get more duplicates with these changes; I hope it works for you too! Let me know.

Looks like tests for the API & ingestion server are failing on this

Looking at this code line in the same Ingestion Server makes me guess the credentials are taken from the environment variables and nothing else is needed in production to connect to S3, but I can't guarantee it. Let me know your thoughts on this approach.

The client authenticates using the instance profile. The instance profile is not configured with permissions for s3, so this change will fail when deployed unless those permissions are added to the ingestion server IAM policy. You'll need to add those new policies to this Terraform resource block: https://github.com/WordPress/openverse-infrastructure/blob/610a8207a50c08e38955bdf069ee0e721ab630f1/modules/services/ingestion-server/iam.tf#L26
Please also add the new bucket in Terraform ensure we control and own the bucket. There's a trend of scanning GitHub for bucket name references and hijacking ones that aren't reserved.

@sarayourfriend That is very helpful, thank you!

If there is a new bucket, please also add the new bucket in Terraform and ensure we control and own the bucket. There's a trend of scanning GitHub for bucket name references and hijacking ones that aren't reserved.

I'm using the existing bucket so there shouldn't be surprises with it :) Good to know that either way!

@AetherUnbound I resorted to simplify the file management given the previous approach was producing strange results in integration tests. This is ready for a re-review now.

It turns out the quotes are needed for tags, or the file creation will raise errors. I undid said change, and locally, the tests passed, so I need to investigate why they aren't working in CI.

Okay, so I just moved the instructions regarding whether the file upload is skipped or not, so there is not much change, which is interesting in relation to tests... Anyway, I'm merging this now!


Variation in GFLang orthography definitions

@simoncozens @moyogo I have been reviewing the GFLang dataset and I am seeing a mix in the way character sets are catalogued.
 Also what is interesting is that the bas_latn also maps out the base/mark pairs that are not precomposed. I really like having that data right in GFLang. As we update GFLang we could make this a consistent practice. Then we could run the no_orphaned_marks check by default and not have to manually add it to a shaperglot profile.
Secondly, we probably should also have the orthographies check look for an auxiliary category and test for those glyphs as well.

It looks these non-precomposed pairs in the ortho list cause the following error when running shaperglot.
  File "/home/neilspatel/.local/bin/shaperglot", line 8, in <module>
    sys.exit(main())
  File "/home/neilspatel/.local/lib/python3.9/site-packages/shaperglot/cli.py", line 97, in main
    options.func(options)
  File "/home/neilspatel/.local/lib/python3.9/site-packages/shaperglot/cli.py", line 47, in check
    results = checker.check(langs[lang])
  File "/home/neilspatel/.local/lib/python3.9/site-packages/shaperglot/checker.py", line 32, in check
    check_object.execute(self)
  File "/home/neilspatel/.local/lib/python3.9/site-packages/shaperglot/checks/orthographies.py", line 30, in execute
    missing = [x for x in self.bases if ord(x) not in checker.cmap]
  File "/home/neilspatel/.local/lib/python3.9/site-packages/shaperglot/checks/orthographies.py", line 30, in <listcomp>
    missing = [x for x in self.bases if ord(x) not in checker.cmap]
TypeError: ord() expected a character, but string of length 4 found```

If we decide to include non-precomposed base/mark pairs in the ortho we need to filter them out for the orthographies check.

OK, while we are waiting for gflang to be better, I'll improve the parsing of decomposed glyphs.

I've changed things so that we can test whether either precomposed or decomposed glyphs are shapable by the font. To do this, I've run the glyph-or-glyphs through Harfbuzz and looked for any .notdefs - this is actually a better test than going through the cmap table, so this is a nice improvement.

Thanks, I'll give it try. That does sound more robust. I guess its more like how you do it in the orphaned marks test.

@simoncozens Does this glyph-or-glyphs do the same as https://github.com/enabling-languages/python-i18n/wiki/icu.CanonicalIterator ?

I don't believe it does check all permutations, but that would be nice rather than having to include all permutations in the exemplar characters.

I guess we really don't anything as complicated as icu.CanonicalIterator, since harfbuzz is going to collapse what it can into precomposed marks. All that is required is getting all permutations of the mark sequence following the base. I just built something in the script I am using to generate the no_orphaned_marks tests that seems to work.
    for basemark in basemarks:
        if len(basemark) > 2:
            base_only = basemark[0]
            marks_only = basemark[1:len(basemark)]
            for i in itertools.permutations(marks_only, len(marks_only)):
                new_basemark = base_only.join(i)
                new_basemarks.append(new_basemark)

In anycase, I don't think we want the all the permutations in gflang. We already have the subsets of all multiple mark combinations in the exemplar_chars list. We should be able to handle all the combinations either directly in the shaperglot check or enumerate them in the shaperglot language profile.

Yeah, I'm being silly. We definitely don't want all permutations in gflang, because if we want decomposition we can handle it ourselves.
I don't even know if it's worthwhile to test all decomposition permutations, given that the first thing Harfbuzz is going to do when it sees the text is normalize it.

I think I need to do a couple tests to be sure. If I recall correctly, for Yoruba e\u0323\u0300 and e\u0300\u0323 has different cluster behavior, which can result in one sequence having an orphaned mark and the other not. Let me confirm this.

That sort of thing is certainly true for syllabic scripts like Myanmar: which is precisely why we don’t want to be throwing every permutation at the shaper - not all of them will be orthographically correct.

That probably means that permutations are best handled specifically in the shaperglot language profile when it is linguistically appropriate and not automatically within the shaperglot check.

@simon, correct me if I am wrong, but the way Shaperglot instantiates HarfBuzz there is no font fall back so we are only looking at the specific fonts being tested.

I think I need to do a couple tests to be sure. If I recall correctly, for Yoruba e\u0323\u0300 and e\u0300\u0323 has different cluster behavior, which can result in one sequence having an orphaned mark and the other not. Let me confirm this.

Ok, I ran this test through shaperglot and printed out the buffers. Test1is e\u0323\u0301 against e\u0301\u0323 and test2 is e\u0323\u0301 against é\u0323
- check: shaping_differs
  inputs:
    - text: "ẹ́"
    - text: "ẹ́"
      language: "ro"
  differs:
    - cluster: 0
      glyph: 0
    - cluster: 0
      glyph: 0
  rationale: "in Yoruba"
- check: shaping_differs
  inputs:
    - text: "ẹ́"
    - text: "ẹ́"
      language: "ro"
  differs:
    - cluster: 0
      glyph: 0
    - cluster: 0
      glyph: 0
  rationale: "in Yoruba"

The output is definitely normalized.
Test1
uni1EB9=0+506|acutecomb=0@-317,0+0 uni1EB9=0+506|acutecomb=0@-317,0+0
uni1EB9=0+506|acutecomb=0@-317,0+0 uni1EB9=0+506|acutecomb=0@-317,0+0

Test2
uni1EB9=0+506|acutecomb=0@-317,0+0 uni1EB9=0+506|acutecomb=0@-317,0+0
uni1EB9=0+506|acutecomb=0@-317,0+0 uni1EB9=0+506|acutecomb=0@-317,0+0

I think that settles the question about permutations.


suggested -h documentation

Thanks Jesse for writing go-flags.
Could I suggest some documentation for handling -h? You mention in the documentation that "Furthermore, the special error type ErrHelp is returned. It is up to the caller to exit the program if so desired." I found that not checking for flags.ErrHelp and just doing an os.Exit(1) resulted in the help being printed twice.
I'm suggesting a sample for the simple case where the programmer just wants to print the help and  exit immediately: https://gist.github.com/soniah/01357b833625b33a0a48
It took me while to work this out (my Go is a bit rusty at the moment) and maybe my code could be improved... Code is from https://github.com/soniah/awsenv

I'm not sure what exactly is not clear about the behavior of -h/--help. It is true that it's a little bit of a strange design in go-flags that the help message is treated as an error, but it sort of works out well. Note that the the documentation of ErrHelp explains that help messages are treated as errors, and that they are automatically printed to stderr if PrintErrors is specified as well. Both these options are enabled in the default case.
The example (in https://github.com/jessevdk/go-flags/blob/master/examples/main.go#L72) also shows how you would write the main parsing in the general case. I.e. you should just exit whenever you get an error and everything should work.


Visual Studio crashes when I quit debugging

I'm running Visual Studio 2013 Community.  I've previously had VS Express 2012 and 2013 (various versions) and tried out VS 2015 Preview and CTP 6 when VS started crashing on me.
Since then I uninstalled all versions of VS, though \Program Files (x86)\Microsoft Visual Studio X.0\ directories exist for 10, 11, 12, and 14 still.  I then installed VS Community 2013 "fresh" (as fresh as it can be with those lingering files).
I installed NTVS for VS 2013 (after uninstalling it, and then removing \Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions\Microsoft\Node.js Tools* manually, since uninstalling did not do it).  Now I can repro this crashing every time with the following steps (before it may have been intermittent?):
Open VS, create a new "Basic Node.js Express 4 Application" project, press F5.  The browser pops up with the expected text.  Close the Browser.  Stop debugging.  VS crashes.
It's quite frustrating.  Until a couple days ago (when I started messing with the Babel package, attempting to get ECMAScript 6 features to work), everything was working great.
Let me know what more information you need.
Thanks!

Since VS crashed, there should be an entry in the Windows event viewer. Mind providing the relevant info?

Any update on this, or are you still blocked?

Sorry for the late reply.  I've just been using a different (inferior) editor to unblock me.
Here's the error that pops up when I get the null reference exception:
Problem signature:
Problem Event Name:	CLR20r3
Problem Signature 01:	devenv.exe
Problem Signature 02:	12.0.31101.0
Problem Signature 03:	54548724
Problem Signature 04:	Microsoft.NodejsTools.Analysis
Problem Signature 05:	1.0.30324.1
Problem Signature 06:	55124335
Problem Signature 07:	6a
Problem Signature 08:	0
Problem Signature 09:	System.NullReferenceException
OS Version:	6.3.96<IP_ADDRESS>.256.103
Locale ID:	1033
Additional Information 1:	5861
Additional Information 2:	5861822e1919d7c014bbb064c64908b2
Additional Information 3:	a10f
Additional Information 4:	a10ff7d2bb2516fdc753f9c34fc3b069
There are two events in event viewer.
First event:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <System>
        <Provider Name=".NET Runtime" />
        <EventID Qualifiers="0">1026</EventID>
        <Level>2</Level>
        <Task>0</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2015-05-13T15:13:46.000000000Z" />
        <EventRecordID>143223</EventRecordID>
        <Channel>Application</Channel>
        <Computer>jordan-pc</Computer>
        <Security />
    </System>
    <EventData>
        <Data>
            Application: devenv.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.NullReferenceException Stack: at Microsoft.NodejsTools.Analysis.AnalysisUnit.AnalyzeWorker(Microsoft.NodejsTools.Analysis.Analyzer.DDG, System.Threading.CancellationToken) at Microsoft.NodejsTools.Analysis.Analyzer.DDG.Analyze(Microsoft.NodejsTools.Analysis.Deque`1<Microsoft.NodejsTools.Analysis.AnalysisUnit>, System.Threading.CancellationToken) at Microsoft.NodejsTools.Analysis.JsAnalyzer.AnalyzeQueuedEntries(System.Threading.CancellationToken) at Microsoft.NodejsTools.Intellisense.VsProjectAnalyzer+AnalysisQueue+GroupAnalysis.Analyze(System.Threading.CancellationToken) at Microsoft.NodejsTools.Intellisense.VsProjectAnalyzer+AnalysisQueue.Worker(System.Object) at System.Threading.ThreadHelper.ThreadStart_Context(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.ThreadHelper.ThreadStart(System.Object)
        </Data>
    </EventData>
</Event>

Second event:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <System>
        <Provider Name="Application Error" />
        <EventID Qualifiers="0">1000</EventID>
        <Level>2</Level>
        <Task>100</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2015-05-13T15:13:47.000000000Z" />
        <EventRecordID>143224</EventRecordID>
        <Channel>Application</Channel>
        <Computer>jordan-pc</Computer>
        <Security />
    </System>
    <EventData>
        <Data>devenv.exe</Data>
        <Data>12.0.31101.0</Data>
        <Data>54548724</Data>
        <Data>unknown</Data>
        <Data><IP_ADDRESS></Data>
        <Data>00000000</Data>
        <Data>c0000005</Data>
        <Data>1298e90c</Data>
        <Data>107c</Data>
        <Data>01d08d8edac5642b</Data>
        <Data>D:\Visual Studio\Community 2013\Common7\IDE\devenv.exe</Data>
        <Data>unknown</Data>
        <Data>a68ab7da-f982-11e4-beee-bc5ff48454c5</Data>
        <Data />
        <Data />
    </EventData>
</Event>

I don't know if it's relevant, but I installed VS to a secondary drive since my primary is a small SSD and rapidly running out space.  I know it shouldn't make a difference, but that doesn't mean it doesn't.

Thx for reporting this. Try deleting your .ntvs_analysis.dat file (renaming is better so we can use it to debug) and re-opening the project to work around the issue for now. Other folks have had good luck with creating a new project from existing, but hopefully you won't have to do that.

Application: devenv.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.NullReferenceException Stack:
at Microsoft.NodejsTools.Analysis.AnalysisUnit.AnalyzeWorker(Microsoft.NodejsTools.Analysis.Analyzer.DDG, System.Threading.CancellationToken)
at Microsoft.NodejsTools.Analysis.Analyzer.DDG.Analyze(Microsoft.NodejsTools.Analysis.Deque`1<Microsoft.NodejsTools.Analysis.AnalysisUnit>, System.Threading.CancellationToken)
at Microsoft.NodejsTools.Analysis.JsAnalyzer.AnalyzeQueuedEntries(System.Threading.CancellationToken)
at Microsoft.NodejsTools.Intellisense.VsProjectAnalyzer+AnalysisQueue+GroupAnalysis.Analyze(System.Threading.CancellationToken)
at Microsoft.NodejsTools.Intellisense.VsProjectAnalyzer+AnalysisQueue.Worker(System.Object) at System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart(System.Object)

In the current code, DDG.Analyze() does not directly call AnalysisUnit.AnalyzeWorker(): https://github.com/Microsoft/nodejstools/blob/35817ffdaa1797f95628a22f588fc3ed885f884b/Nodejs/Product/Analysis/Analysis/Analyzer/DDG.cs#L37. Does this mean user is using an older version of NTVS?

@hoanhtien This is more likely a difference between Debug and Release mode. In Release mode, the compiler performs some extra optimizations that it doesn't perform in Debug mode. In this case, it looks like there was some inlining which caused a missing entry in the stack trace.

This fix is now available in our latest dev build - mind verifying it works on your end?


Sentinel-2 L2A products query with get_area_info

When doing a get_area_info request the way described below, I receive no L2 products. I have a feeling that it used to return both L1 and L2 products before, because I didn't make any change with my code and I used to receive L2 products, but maybe I just remember it wrongly.
import datetime
from sentinelhub import CRS, get_area_info, BBox
bb = BBox('14.1575,49.8967,14.7763,50.2224', CRS.WGS84)
get_area_info(bb, (datetime.date(2018, 4, 7), datetime.date(2018, 5, 10)), 20)

Is the change of the behaviour described possible? In the case I am wrong and no change has caused this, is there any similar way to get the info also for L2 products covering the area?
Using sentinelhub==3.4.0.

Hi @pesekon2 ,
OpenSearch implementation only works with Sentinel-2 L1C products.
In order to get L2A information, you can use SentinelHub's CatalogAPI (to get tiles of any collection imported with SentinelHub):
import datetime
from sentinelhub import CRS, BBox, SentinelHubCatalog, DataCollection

catalog = SentinelHubCatalog()

bb = BBox('14.1575,49.8967,14.7763,50.2224', CRS.WGS84)
time_interval = time=(datetime.date(2018, 4, 7), datetime.date(2018, 5, 10))

searcher = catalog.search(bbox=bb, time=time_interval, collection=DataCollection.SENTINEL2_L2A)
info = list(searcher)

Hi @batic,
Thanks, I'll try to follow this way then.
But if I understand [1] correctly, I actually cannot do this without an account on sentinelhub, because I have to configure the SentinelHubCatalog() object, right? So every user of the tool would need a sentinelhub account extra to the AWS one. Or is there any way to do it without this account?
[1] https://sentinelhub-py.readthedocs.io/en/latest/configure.html#sentinel-hub-configuration

You are right, Sentinel Hub account is needed for using Catalogue service. There is currently no alternative option to automatically query L2A data on this AWS repository.
Note that there are trial accounts available for Sentinel Hub, which should allow you to make up to 100.000 queries over a period of a month.

Okay, thanks for the quick reply. I'll go for that.


Split the node network layer out of the node kernel

This should be a pure refactoring, with no changes. It takes the network interface initialisation out of the NodeKernel and moves it into its own layer which can be constructed from the NodeKernel.
This is another step in the direction of fully integrating the remaining network layer features (mux, version negotiation etc).

bors r+


refactor(model, cache, gateway): update bitflags crate to v2

Updates bitflags to 2. Bitflags crate changes: https://github.com/bitflags/bitflags/blob/main/CHANGELOG.md#whats-changed-5

Actually this is a breaking change. Please retarget this PR to next.

Why is this a breaking change?

Could you rebase the PR so it only includes your new commits?

Could you rebase the PR so it only includes your new commits?

done. (I think haha)


Error when trying to consume a topic with NULL values

When running a local test in my dev schema after upgrading the Kafka Connector to 1.0.0, I ran into the following issue:
[main] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-EXASOL_KAFKA_UDFS_CONSUMERS-1, groupId=EXASOL_KAFKA_UDFS_CONSUMERS] Fetch position FetchPosition{offset=117013, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[<IP>:<PORT> (id: 22 rack: null)], epoch=0}} is out of range for partition public.pro-seller.offer-0, resetting offset
[main] INFO org.apache.kafka.clients.consumer.internals.SubscriptionState - [Consumer clientId=consumer-EXASOL_KAFKA_UDFS_CONSUMERS-1, groupId=EXASOL_KAFKA_UDFS_CONSUMERS] Resetting offset for partition public.pro-seller.offer-0 to position FetchPosition{offset=139566, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[<IP>:<PORT> (id: 22 rack: null)], epoch=0}}.
[main] INFO org.apache.kafka.common.metrics.Metrics - Metrics scheduler closed
[main] INFO org.apache.kafka.common.metrics.Metrics - Closing reporter org.apache.kafka.common.metrics.JmxReporter
[main] INFO org.apache.kafka.common.metrics.Metrics - Metrics reporters closed
[main] INFO org.apache.kafka.common.utils.AppInfoParser - App info kafka.consumer for consumer-EXASOL_KAFKA_UDFS_CONSUMERS-1 unregistered
F-UDF-CL-SL-JAVA-1080: Exception during run
com.exasol.cloudetl.kafka.KafkaConnectorException: Error consuming Kafka topic 'public.pro-seller.offer' data. It occurs for partition '0' in node '0' and vm '140681706548736' Cause: null
com.exasol.cloudetl.kafka.KafkaTopicDataImporter$.run(KafkaTopicDataImporter.scala:99)
com.exasol.cloudetl.kafka.KafkaTopicDataImporter.run(KafkaTopicDataImporter.scala)
com.exasol.ExaWrapper.run(ExaWrapper.java:196)
Caused by: java.lang.NullPointerException
com.exasol.cloudetl.kafka.KafkaTopicDataImporter$.$anonfun$run$1(KafkaTopicDataImporter.scala:84)
com.exasol.cloudetl.kafka.KafkaTopicDataImporter$.$anonfun$run$1$adapted(KafkaTopicDataImporter.scala:71)
scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
scala.collection.AbstractIterable.foreach(Iterable.scala:919)
com.exasol.cloudetl.kafka.KafkaTopicDataImporter$.run(KafkaTopicDataImporter.scala:71)
	... 6 more

W-UDF.CL.SL.JAVA-1082: Skipping init, because cleanup method cannot be found.

This seems to relate to the following line in KafkaTopicDataImporter.scala:
          val exasolRow: Seq[Any] = recordValue ++ metadata

Is there a possibility that the above construct will fail if the recordValue is null? This is something that quite frequently happens in the topic I'm reading here, as a way to indicate for a record to be deleted..
Previously, that particular line looked like:
          val exasolRow: Seq[Object] = getAvroRow(singleColumnJson, recordValue) ++ metadata

..where getAvroRow() was defined like:
  private def getAvroRow(singleJson: Boolean, recordValue: GenericData.Record): Seq[Object] =
    if (singleJson) {
      Seq(s"$recordValue")
    } else {
      AvroRow(recordValue).getValues().map(_.asInstanceOf[AnyRef])
    }

Apparently the null conversion would pass by just fine as part of the string interpolation above (I'm extracting value as JSON).

@jwarlander, thanks for reporting this issue. @morazow is available again next Monday and will look into the matter then.

Great, thanks @redcatbear! Sadly, I guess I'll have to roll back to using the previous version in the meantime -- I did have a look myself, but wasn't able to set up a proper test for pushing in a null value. Scala isn't a commonly used language for me unfortunately :sweat_smile:

As it turns out, I was able to implement what is at least a workaround.. see #31. This is only tested with our use case so far though, single column JSON extraction from Avro-encoded messages.

@morazow, happy for any feedback here.. especially around how to test this! :sweat_smile:

Hello @jwarlander,
Thanks for the feedback!
I am going to check your pull request, maybe we could have a new minor release with the fix.


Bump version numbers for v0.1.0

What this PR does / why we need it:
Bumps version numbers in manifests for v0.1
Release note:
NONE

/retest


(#92) - Fix Travis

(hopefully ;)

Don't mean to be silly but does this even require a +1?

@yaronyg On repo's that aren't pouchdb/pouchdb we sometimes just push if a PR stands unopposed after a few (say 3) days. pouchdb/pouchdb always needs a +1.

@marten-de-vries - Thanks for reminding me, @daleharvey had told me that but I forgot. I'm guessing however that it is 'polite' if one has actually looked at the commit to +1 anyway just so the commiter doesn't have to wait a few days?
BTW, I'm asking these annoying questions because I'm new to the community and it's easier to ask and be slightly annoying than to do it wrong and seriously piss someone off. :)

@yaronyg It's no bother at all. To answer your question: yes, a +1 is always appreciated, I consider timebombing (pushing to master without a +1) pretty much a last resort.

Closing in favour of https://github.com/pouchdb/pouchdb/pull/3373.
@yaronyg This is a good example of why we review even small commits, there was an easier way available I didn't know about in this case.


Add alert to python rest api

skip skydive-go-fmt
skip skydive-functional-tests-backend-elasticsearch
skip skydive-functional-tests-backend-orientdb
skip skydive-scale-tests
skip skydive-cdd-overview-tests
skip skydive-unit-tests
skip skydive-compile-tests
skip skydive-k8s-tests
skip skydive-ppc64le-tests

Thank you @SH118415
Could you please also add tests for this ?

run skydive-python-tests

run skydive-python-tests

@SH118415 You can call pep8 and flake8 locally to avoid most of the Python syntax errors.
You can also run the python test suite using this https://github.com/skydive-project/skydive/blob/master/scripts/ci/run-python-tests.sh#L11-L24

@SH118415 You can call pep8 and flake8 locally to avoid most of the Python syntax errors.

@SH118415 Can you please squash the 2 commits into 1 ?

Done

run skydive-python-tests


Update aws-java-sdk-s3 to 1.12.728

About this PR
📦 Updates com.amazonaws:aws-java-sdk-s3 from 1.12.726 to 1.12.728
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.amazonaws", artifactId = "aws-java-sdk-s3" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "com.amazonaws", artifactId = "aws-java-sdk-s3" }
}]

labels: library-update, early-semver-patch, semver-spec-patch, commit-count:1

Superseded by #683.


Make recipe noarch

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[n/a] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[n/a] Ensured the license file is being packaged.

Use https://github.com/conda-forge/conda-forge.github.io/pull/1839 and https://github.com/conda-forge/constructor-feedstock/pull/63 as reference.
The linter message can be ignored because it is now possible to have selectors with noarch package.

Yes, with this PR, the installation instruction can be simplified.
This is currently holding new releases of the hyperspy bundle because kikuchipy added
pyebsdindex-base as dependency and the packages for osx-arm64 are missing.

This is currently holding new releases of the hyperspy bundle because kikuchipy added pyebsdindex-base as dependency

Aha... Estimation of orientations and projection centers via Hough indexing in PyEBSDIndex is instrumental to the usefulness of kikuchipy. Again, thank you for taking the time to help with the release of PyEBSDIndex.

This noarch approach is quite new but I have tested the new package on various platform (win, linux, osx_arm64) and from what I can tell by running the pyebsdindex test suite the change of this PR seems to work as expected! :)
This is quite useful because it would save the migration to other platforms or new python versions.


tests: bump dd-trace-java

What does this PR do?
Align with https://datadoghq.dev/ciapp-alpha-docs/tests/java#recommended-configuration
Additional Notes

Review checklist
Please check relevant items below:

[ ] This PR includes all newly recorded cassettes for any modified tests.

[ ] This PR does not rely on API client schema changes.

[ ] The CI should be fully passing.

[ ] Or, this PR relies on API schema changes and this is a Draft PR that includes tests.


Update Start-SPOTenantRename.md

Added not available for GCC, GCC High & DoD.

@MSFT-GRRojas Gentle reminder. Thanks

@tiburd Please merge this PR. Thanks


<EMAIL_ADDRESS>

<EMAIL_ADDRESS>

Hello @bazelbuild/bcr-maintainers, modules without existing maintainers (boost.exception) have been updated in this PR. Please review the changes.

Need skip-url-stability-check label


Ingest and display NIBRS data

addresses #18
Some notes:

while this ingests all offense-level fields from the NIBRS data, it doesn't display all of them in the "NIBRS Offenses" table on the incident page - I wasn't sure what would be most useful and figured we can easily add more.
for incidents that don't exist in the BPD's crime incident data, I don't try to construct a value for occurred_on_date from the date/time-related columns in the NIBRS data. This is probably doable, though - maybe something for a future PR!

Everything looks good to me! I opened another issue for the date/time stuff #22. And, yes, we can iterate on how the data is displayed, no reason to block on that.
I'll merge the PR and deploy it.


Fix bug when infos are missing and make lists more touchfriendly

Fixes a bug, when for example adressdata was missing, the detailpage was not opened.
Also has some list design and loading indicators.

Thanks :)

Really nice app ;-) I have really missed a veggie app till now!


Pull-to-refresh on unreads screen and streams/PMs screens

We often have bugs with the symptom that the app doesn't update its view of what messages the user has gotten. This can be frustrating, but as a user what really makes it frustrating rather than mildly annoying is that there's nothing you can do about it, short of force-closing the app and reopening. This has been a recurring complaint from our most active users for basically as long as the app has existed.
In many apps like Gmail, the presented information (e.g., the emails in your inbox) may update on its own but also the user can force a refresh right now, with a pull-to-refresh gesture -- when scrolled to one end of the list, pull as if to scroll further. The app refreshes the info, and a spinner appears while it does so.
We should borrow the same feature.
Ideally the experience we would like to present would make this unnecessary -- the user would see the app automatically keep itself up to date on all the information it's presenting, and would never need this. In the past we've rejected requests for this feature (#1473; #1611 if I'm correctly interpreting that issue; many discussions in chat), often on these grounds. Empirically we have not lived up to that promise, and we should give the user a reasonable tool to handle that.
This issue is for providing this feature on the screens of the main nav: the unreads screen, and the streams and PMs screens. A second issue will track providing this feature on the message list, which is likely to work differently.

@zulipbot claim

I still think that the UX will be extremely confusing and might not result often (if ever) in an actual refresh of data.


[chore] Split requirements and freeze

Add a file requirements.in which specifies what we want and use pip-compile to build requirements.txt which freezes what we know works
https://github.com/jazzband/pip-tools
Alternative to consider https://github.com/astral-sh/uv

Appreciated PR! I had a similiar branch locally but left it for better days to deduct direct and indirect dependencies. It is a good start. Would you mind if you already put a README.md instructions for pip compile?

It is "documented" in the requirements.txt
pip-compile requirements.in

https://github.com/opengisch/QFieldCloud/pull/1010/files#diff-6d1d18ca0d67cd16278085ea3947f2f19f91acf1b388e3534f46f561f03b1361R5
So I think eventually anyone in the desire to work on dependencies will eventually run into that. Do you think it's worth adding more docs related to that?
FTR: I also considered a pre-commit integration but we don't want to slow commits to 30 seconds. I think it needs to be on-demand on the dev machine. It could be integrated into a CI check.

AI think we should get rid of (many of the) fixed version numbers in the .in file, as they are mostly a relict of the old frozen requirements.
Apart from django itself, are there other consciously pinned deps?

I contininuosly push them to newer versions. I am pretty sure AI would suggest we bump to Django 5, would love to see it upgrading the codebase :) . Let's keep the numbers and reduce the direct dependencies.

Ok, I think this is the final push to this PR for now. I think the end result is quite good, however there is a follow up task to create a CI test that the developer has pushed their requirements.in and requirements.txt changes. I would let this to be a future onboarding task for someone :)


Django: Load templates from subdirectories

Hi!
I have a need that templates should be loaded from the subdirectories as well. Currently, the implementation of django.LoadDirectory just processes the current template folder. Would you be willing to accept a pull request implementing this functionality?
On a related note, I'd probably better error reporting for errors which occur in the templates.

Oh, nevermind. This is actually a problem with filepath.Walk as my situation contains a symlink which does not get traversed. The django API support directories just fine :)


Robust standard errors

Can ZAM be used with alternative derivations of standard errors? robust (HC1,2,3...), cluster, bootstrap, etc.?
If so, how?

zaminfluence can be easily used for clustered standard errors.  Just pass in the clustering index to the se_group argument in ComputeModelInfluence.  (See this line in the examples).
If you're willing to hack, it'd be easy in principle to make different choices in the code for computing standard errors.  As long as the computation is differentiable, torch will take care of the derivatives for you.  The code you'd change is here.
The bootstrap would require more work, but is doable in principle.  You'd have to say what it meant for the bootstrap samples if you had "left out" a datapoint from the original dataset, though.  For example, if a datapoint were resampled three times in a particular sample, but it were left out of the original dataset, how would your bootstrap have changed?  But the core idea --- express your quantity of interest as a differentiable function of data weights --- can apply.


modified the lisence

Ⅰ. Describe what this PR does
this pr is to modify the lisence to finish task 111
Ⅱ. Does this pull request fix one issue?

fixes #XXXX
Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews

/lgtm


Fix alternative os builds

added docker failed check to and skip pytest decorator to tests requiring it
decoupled dependence on environment driver failure for modules that don't require it.
refactored docker environment driver code to include separate init and init() functions

refactored code to work on windows and skip docker tests if not initialized or linux images not buildable.


Node Attribute Lifting

This method involves lifting a graph to a hypergraph based on node attributes. Hyperedges are created by grouping nodes with the same attribute. Given (n) attributes of a node, users can choose which attribute to use for constructing the hyperedges. Additionally, users can preprocess the data to add new attributes for grouping. For example, in a social network, hyperedges can represent people from same city, or nodes can be grouped based on domain-specific distances.
This lifting approach is particularly useful for datasets where node attributes are present. For instance, the MUTAG, ENZYMES, and PROTEINS datasets all have this property. We tested our method on these datasets and updated the tutorial accordingly.

Hello @demiqin  ! Thank you for your submission. As we near the end of the challenge, I am collecting participant info for the purpose of selecting and announcing winners. Please email me (or have one member of your team email me) at<EMAIL_ADDRESS>so I can share access to the voting form. In your email, please include:

your first and last name (as well as any other team members)
the title of the method you implemented
the input domain of the method you implemented
the output domain of the method you implemented
your pull request number (#40)

Before July 12, make sure that your submission respects all Submission Requirements laid out on the challenge page. Any submission that fails to meet this criteria will be automatically disqualified.


Add reorder_attributes config to sort doc attributes before other attributes

This was discussed in https://github.com/rust-dev-tools/fmt-rfcs/issues/72 with the conclusion that the style guide recommends doc comments before attributes, with an option in rustfmt to do the reordering that is off by default.
I did not find an existing option that does this, nor an issue to track, so here we go. :)
Expected diff:
- #[cfg(cfg)]
  /// doc
+ #[cfg(cfg)]
  struct S;

I think this issue and #3744 are the same. Even though this came first, #3744 has more active discussion. I'd recommend closing this @dtolnay.

Going to close this in favor of https://github.com/rust-lang/rustfmt/issues/3744


not enough params

Block 14 of the notebook:
train_imgs  = parse_annotation(coco_train_path)
which uses parse_annotation from the python preprocessing.py script should be given a second param 'labels'.
TypeError: parse_annotation() missing 1 required positional argument: 'labels'

@GiladAmar thanks! It's fixed.

Thank you, that does the trick:-)


[Event Grid] EventGridPublisherClient #send (CloudEvent schema) enriches events with distributed tracing information failing in nightly runs

Event Grid nightly test runs are failing with:

Stack trace:
AssertionError: expected 'HTTP POST' to equal '/api/events'
at Context. (test/public/eventGridClient.spec.ts:260:15 <- dist-test/index.browser.js:43244:23)

For more details check here:

https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1304688&view=results
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1307982&view=results

I'll take this, it's related to #17973


bio.h: fix typo in BIO_get_conn_int_port() macro

Regression from: https://github.com/openssl/openssl/commit/2a60fccdd9b696e01fddaa268e92ea210beb0e8f

thanks for your diligence.  already noted and being updated now.

Thank you!


Fix deploy_blockchain_incremental_gcp.sh for tracker

Fix deploy_blockchain_incremental_gcp.sh for tracker

Thanks for the review. Now merging..


Uninstall documentation

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3000
Originally created by @matt535 on 2021-10-20 19:53:31

Problem to solve
Need to make it clearer that the way to force a reinstall of a tap is to delete the directory in the .meltano directory.
Currently there is documentation on how to install a custom plugin, but especially when using custom taps, it isn't immediately clear how to remove them.
Target audience
Developers building custom taps and using meltano as part of the development process.
Further details
This currently is relatively undocumented, only in Slack is it made clear that removing files in .meltano/extractors will trigger a complete reinstall, ie once deleted, meltano install <type> <name> --reinstall will reinstall it.
Proposal
Provide a proper reinstall option, ie meltano install <type> <name> --reinstall, with the same impact of a `rm -rf .meltano/extractors/tap-
Currently in the docs it indicates that reinstall is possible, but this does not do a complete reinstall.

To (re)install a specific plugin in your project, use meltano install  , e.g. meltano install extractor tap-gitlab.

What does success look like, and how can we measure that?
Able to remove a tap without deleting files in .meltano
Links / references

Install docs
Slack conversation

Please note that this was taken from GitLab, to be changed accordingly

View 6 previous comments from the original issue on GitLab


Target Portal and iqn are missing in JSON output when VSM is in pending state and evalstatus is complete.

The following is the output when a create volume command is issued with VSM in pending state.
Target Portal and iqn fields are missing.
{
   "Spec":{
      "AccessModes":null,
      "Capacity":null,
      "ClaimRef":null,
      "OpenEBS":{
         "volumeID":""
      },
      "PersistentVolumeReclaimPolicy":"",
      "StorageClassName":""
   },
   "Status":{
      "Message":"",
      "Phase":"",
      "Reason":"complete"
   },
   "annotations":{
      "evalpriority":"50",
      "evaltype":"service",
      "evaltrigger":"job-register",
      "evaljob":"demo-vsm1-vol1",
      **"evalstatus":"complete"**,
      "evalstatusdesc":"",
      "evalblockedeval":"0689254c-62ca-84a6-1fdb-caf74f7d46a0"
   },
   "creationTimestamp":null,
   "name":"demo-vsm1-vol1"
}

Targetportal & iqn will be provided in info requests only.
Consumers need to issue info API request.


Can not hide the close button !

I want to hide the close button . i used this but it doesn't work
this.mydockcontent.CloseButtonVisible = false ;

Which theme are you using?  A quick test in the DockSample has this working with everything except the 2003 theme (which should probably be addressed).

i used new blue theme . can u send me a sample project that u using CloseButtonVisible ?, i will try run it on my local .
Thanks  so much !

In the sample application included in the solution, I opened DummyOutputWindow in the Designer and change CloseButtonVisible to false.  This worked as expected with the 2013 theme.

Have to disagree
Does not work for me.
I have my dockpanel set up as DockingWindow, adding 2 DockContents with DockState.Document, but CloseButtonVisible does nothing.

in DrawTab_Document (VS2012LightDockPaneStrip.cs) you do not check for visibility at all

Same issue Vs2012LightTheme

Related to #194. To shed some light here with a long history. The original property CloseButtonVisible was designed for old themes VS2003 and VS2005, where there is a dedicate close button for all tabs.
When I worked on the new theme of VS2012, I decided not to use this property for the new theme, as there is a close button per tab. Thus, this behavior was desired.
If you do need to hide close buttons in VS2012/2013 themes, please derive from them your own themes and then remove the button rendering code.
It is a design goal to make each themes matching their corresponding VS look and feel, so I don't think we should left switches for visual elements everywhere to pollute the code base.

if CloseButtonVisible only for vs2003\vs2005 theme remove it from new theme or add label like deprecated or something else. I think if object has setter CloseButtonVisible it's must work.

@verysimplenick if you run your own open source project, you will see how difficult a choice like that can be made. This thread has been marked as FAQ candidate and that's so far the thing I can do.


Transactions are commited in CommitRequest.NON_TRANSACTIONAL mode

We have just been bitten by this using the latest 0.5.0.
We had the following piece of code, run in two different threads:
with dataset.transaction as t:
  entity = dataset.get(entity_key)
  archived_entity = _move_to_archive_key(entity)
  t.delete(entity.key)
  t.put(archived_entity)
  return archived_entity

We've seen two copies of the archived_entity for the same entity.  We've managed to reproduce it with two threads performing this concurrently (running separate Connections each, because httplib2 is not thread safe).
Having used Datastore's underlying technology before, I expected to get a ConcurrentModificationException on delete, and one of the threads throwing it, similar to what the Java API for AppEngine describes:
https://cloud.google.com/appengine/docs/java/datastore/transactions#Java_Uses_for_transactions
However, both of them succeed creating two copies of archived_entity, and there's no ConcurrentModificationException anywhere in gcloud-python code.
We've tracked down to the following:

Transaction#commit calls the Batch#commit https://github.com/GoogleCloudPlatform/gcloud-python/blob/master/gcloud/datastore/transaction.py#L177
Batch#commit is unaware of the Transaction#id, calls Connection#commit without the transaction_id parameter https://github.com/GoogleCloudPlatform/gcloud-python/blob/master/gcloud/datastore/batch.py#L214
Connection#commitgets transaction_id, and sets the REST request's mode to datastore_pb.CommitRequest.NON_TRANSACTIONAL in https://github.com/GoogleCloudPlatform/gcloud-python/blob/master/gcloud/datastore/connection.py#L321

Impact: This means that all transactions made through python-gcloud are non-transactional. Please treat it as a P0 bug.

We managed to get the correct behaviour by plumbing through Transaction#id through Batch#commit. The result is that the above test code works as intended: one thread doest the complete transaction, the other one gets:
 File "~/src/python/improbable/db/common/transaction.py", line 15, in commit
   response = self.connection.commit(self._dataset_id, self.mutation, self._id)
 File "~/virtualenv/python/local/lib/python2.7/site-packages/gcloud/datastore/connection.py", line 324, in commit
   datastore_pb.CommitResponse)
 File "~/virtualenv/python/local/lib/python2.7/site-packages/gcloud/datastore/connection.py", line 108, in _rpc
   data=request_pb.SerializeToString())
 File "~/virtualenv/python/local/lib/python2.7/site-packages/gcloud/datastore/connection.py", line 85, in _request
   raise make_exception(headers, content, use_json=False)
Conflict: 409 too much contention on these datastore entities. please try again.

I think the right solution here would be to catch it in the Transaction#commit and rethrow it as ConcurrentModificationException.

What's the ETA on a version that supports it? :)

@mwitkow-io I'm going to cut a release today (or tomorrow).

RE: "opportunistic concurrency model of Datastore is poorly exposed to the end user." we expect the error status codes and error messages from the Datastore API (the RPC service) to be useful. If there are issues where they aren't useful, we should take it up with the API folks.
The ConcurrentModificationException in App Engine corresponds to one of these RPC statuses.

@mwitkow-io Also I'd love to chat about lack of thread-safety in httplib2 and how you address it. Want to hop on a video chat?
We actually don't require use of it, but this is not at all clear from our docs (and may never be).

@mwitkow-io Also I'd love to chat about lack of thread-safety in httplib2 and how you address it. Want to hop on a video chat?
We actually don't require use of it, but this is not at all clear from our docs (and may never be).

Sure thing, would be great to chat!
I'm in London though, so 8h diff to MTV. 5pm tomorrow BST? Please send a
call invite :-)
On Thu, 4 Jun 2015 18:47 Danny Hermes<EMAIL_ADDRESS>wrote:

@mwitkow-io https://github.com/mwitkow-io Also I'd love to chat about
lack of thread-safety in httplib2 and how you address it. Want to hop on
a video chat?
We actually don't require use of it, but this is not at all clear from our
docs (and may never be).
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/gcloud-python/issues/903#issuecomment-108989718
.

Invite sent for 4pm your time (if you're free). I assumed you use the email listed on your GitHub profile. If 4pm doesn't work I gave you edit privileges on the event.

Thanks, that works great :-)
On Thu, 4 Jun 2015 19:16 Danny Hermes<EMAIL_ADDRESS>wrote:

Invite sent for 4pm your time (if you're free). I assumed you use the
email listed on your GitHub profile. If 4pm doesn't work I gave you edit
privileges on the event.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/gcloud-python/issues/903#issuecomment-108996945
.


How to upgrade your flutter project (app) to the latest flutter version after null safety?

I want to upgrade my project which is on Flutter 2.0 to the latest version of flutter (2.5).
I have already seen this question on Stackoverflow which is most relevant.
How do I upgrade an existing Flutter app?
In my Mac, I have Flutter upgraded to 2.5. You can see the output of flutter doctor -v below:
[✓] Flutter (Channel stable, 2.5.0, on macOS 11.5.2 20G95 darwin-arm, locale
    en-PK)
    • Flutter version 2.5.0 at /Users/ahmad/development/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 4cc385b4b8 (9 days ago), 2021-09-07 23:01:49 -0700
    • Engine revision f0826da7ef
    • Dart version 2.14.0

[✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at /Users/ahmad/Library/Android/sdk
    • Platform android-31, build-tools 31.0.0
    • Java binary at: /Applications/Android
      Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7249189)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.5.1, Build version 12E507
    • CocoaPods version 1.11.0

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2020.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7249189)

[✓] Connected device (2 available)
    • iPhone 12 Pro Max (mobile) • 57868E1C-4871-4185-A797-383121736AB8 • ios
      • com.apple.CoreSimulator.SimRuntime.iOS-14-5 (simulator)
    • Chrome (web)               • chrome                               •
      web-javascript • Google Chrome 93.0.4577.82

• No issues found!

I have searched over the internet about it. Mostly, I see two things, either how to upgrade to null-safety (which means from earlier versions to 2.0) or how to upgrade Flutter sdk in your system.
I have tried running flutter upgrade command which says I'm up-to-date.
I have also tried running flutter pub upgrade.
Both do not change the version of flutter in the pubspec.lock file inside my project which says at the end of file:
sdks:
  dart: ">=2.14.0 <3.0.0"
  flutter: ">=2.0.0"

Please guide me how can I upgrade my project to the latest version.
Thanks in advance :)

Hi @rmahmadkhan
You simply need to change SDK constraints to 2.12.0 and above in your pubspec.yaml to the following
environment:
  sdk: ">=2.12.0 <3.0.0"

When you create a new project on the latest stable, you will same the SDK constraints, new projects have null-safety enabled by default.
After that, you might need to check packages and upgrade to null-safety release, visit package's pub page for such information
This issue doesn't seem to describe a bug or a feature request. Please see https://flutter.dev/community for resources and asking questions like this, you may also get some help if you post it on Stack Overflow and if you need help with your code, please see https://www.reddit.com/r/flutterhelp/
Closing, as this isn't an issue with Flutter itself. If you disagree, please write in the comments and I will reopen it.
Thank you


Token mold is not showing up in the Configure Modules tab

I have token mold enabled and it's working, I just can't configure anything. TM seems to not be showing up at all in the configuration settings of the system settings. I don't know if there's something I'm doing wrong, but I've removed all modules except token mold and it is not showing up.

All Token Mold settings can be found at the top of the "Actors Directory" tab.


Error: TypeError: buf.copy is not a function

Just working on a fresh project using socketcluster create lab1 and getting loads of errors in the server output
1454070786007 - Origin: Balancer
    Error: TypeError: buf.copy is not a function
    at Function.Buffer.concat (buffer.js:239:9)
    at IncomingMessage.<anonymous> (/path/to/lab1/node_modules/socketcluster/lib/scbalancer.js:306:31)
    at emitNone (events.js:72:20)
    at IncomingMessage.emit (events.js:166:7)
    at endReadableNT (_stream_readable.js:905:12)
    at nextTickCallbackWith2Args (node.js:474:9)
    at process._tickDomainCallback (node.js:429:17)
   >> Balancer PID: 2779

Am on node 5.4.1 if that helps

@poppahorse Which version of SC are you using? Version 1.x.x?

Sorry should have mentioned that too. Freshly installed via npm so 1.3.2

Closing. Dropping support for version 1 of SC.

@poppahorse Ran into the same problem. Switch to node v0.10.44 (use nvm if you like) and seems fine now. 4.2 had same problem as 5.x.


[Function Expose] Table 5847 - Average Cost Calc. Overview

We need functions "CalculateRemainingQty", "CalculateCostAmt" and "CalculateAverageCost" availables as global/external.

Thank you, this functions will be exposed in the future release.


feature: Noncommuting observables when shots=0

Allows arbitrary observables to be attached to a circuit, allowing, for example, the simultaneous calculation of all terms of a qubit Hamiltonian when shots=0.
Issue #, if available:
Description of changes:
Testing done:
Merge Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.
General

[x] I have read the CONTRIBUTING doc
[x] I used the commit message format described in CONTRIBUTING
[x] I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

[x] I have added tests that prove my fix is effective or that my feature works (if appropriate)
[x] I have checked that my tests are not configured for a specific region or account (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Allows arbitrary observables to be attached to a circuit, allowing, for example, the simultaneous calculation of all terms of a qubit Hamiltonian when shots=0.

Clarify this a bit more - it should be clear what happens. Namely, the user can create a circuit with arbitrary observables, but an error is returned if it is run with shots > 0.


IVS-64 - CLS000 - Classification Association

Continuation of https://github.com/buildingSMART/ifc-gherkin-rules/pull/245
Relevant CT https://ifc43-docs.standards.buildingsmart.org/IFC/RELEASE/IFC4x3/HTML/concepts/Object_Association/Classification_Association/content.html
With the new CI-CD testing for activation;

On a local deployment of the service, the passing file;

Na-files

All files are syntax- and schema compliant

cc @evandroAlfieri

Nice work!  I tested with an example model from the recent buildingSMART US workshop and CLS000 correctly activated as expected.


grpc/transcoding: Add support for google.api.http_body response streaming

Description: Add support for httpBody for response streaming use case.
Risk Level: Low
Testing: New unit / integration test
Docs Changes: Added
Release Notes: Added
Fixes #5540
This is my first contribution to Envoy project, so I need your help to ramp up with code base.
Please check inline questions and help resolve them.

BTW, what is the preferred way to resolve conflicts?
Is it OK to rebase once you've done with review or merge is a way to go?

BTW, what is the preferred way to resolve conflicts?
Is it OK to rebase once you've done with review or merge is a way to go?

Please check https://github.com/envoyproxy/envoy/blob/master/CONTRIBUTING.md#submitting-a-pr.
TL;DR Please only merge, do not rebase.

All done! :)

@lizan please let me know if something else needs to be done to get this merged.. )

@lizan ping?

Sorry we need another master merge to fix conflict in version history. Thanks

Sorry we need another master merge to fix conflict in version history. Thanks

Resolved! :)

it seems that coverage test failed.

Please merge master (this should also fix coverage).
/wait

Please merge master (this should also fix coverage).
/wait

Sure, merged and fixed
All checks are now done!

@lizan ping

@belyalov I tested this pr to transcode a streaming grpc to chunked http response, but the content-type of curl is not correct.
curl "localhost:8080/get?filename=testdata/music.mp3" -4 -v > test_chunk_music.mp3
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying <IP_ADDRESS>...
* Connected to localhost (<IP_ADDRESS>) port 8080 (#0)
> GET /get?filename=testdata/music.mp3 HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json
< grpc-accept-encoding: identity,deflate,gzip
< accept-encoding: identity,gzip
< x-envoy-upstream-service-time: 0
< date: Thu, 23 Apr 2020 14:15:00 GMT
< server: envoy
< transfer-encoding: chunked
<
{ [14238 bytes data]
100 1999k    0 1999k    0     0   640k      0 --:--:--  0:00:03 --:--:--  639k^C

If I only send the chunk of data once, the content-type is correct.
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying <IP_ADDRESS>...
* Connected to localhost (<IP_ADDRESS>) port 8080 (#0)
> GET /get?filename=testdata/musc.mp3 HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: audio/mp3
< grpc-accept-encoding: identity,deflate,gzip
< accept-encoding: identity,gzip
< x-envoy-upstream-service-time: 0
< date: Thu, 23 Apr 2020 14:18:19 GMT
< server: envoy
< transfer-encoding: chunked

Is that the subsequent response chunk overwrite the content-type set by the first chunk?

hmm,  this line
should return StopIteration.
You can try it.

hmm, this line
should return StopIteration.
You can try it.

Hm, it does return:
https://github.com/envoyproxy/envoy/blob/62777e87edc9f16af5c7920bb86f198585789a3d/source/extensions/filters/http/grpc_json_transcoder/json_transcoder_filter.cc#L503-L511

@npuichigo  could you debug it with "-l debug" log?

@npuichigo could you please verify that you're using latest version from master?
I've tried to reproduce it on my test setup, it seems works well:
Having golang as gRPC server
func (k *KV) GetHttp(in *kv.GetRequest, stream kv.KV_GetHttpServer) error {
	log.Printf("get: %s", in.Key)

	body := &httpbody.HttpBody{
		ContentType: "text/plain",
		Data:        []byte("data\n"),
	}
	stream.Send(body)
	body.ContentType = "unknown"
	stream.Send(body)

	return nil
}

Envoy configured with default gRPC transcoding:
#...
          - name: envoy.filters.http.grpc_json_transcoder
            typed_config:
              "@type": type.googleapis.com/envoy.config.filter.http.transcoder.v2.GrpcJsonTranscoder
              proto_descriptor: "kv.pb"
              services: ["kv.KV"]
              print_options:
                add_whitespace: true
                always_print_primitive_fields: true
                always_print_enums_as_ints: false
                preserve_proto_field_names: false
#...

Request / Response:
$ curl -v http://localhost:8080/getHttp
*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 8080 failed: Connection refused
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to localhost (<IP_ADDRESS>) port 8080 (#0)
> GET /getHttp HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: text/plain
< x-envoy-upstream-service-time: 1
< date: Fri, 24 Apr 2020 01:17:11 GMT
< server: envoy
< transfer-encoding: chunked
<
data
data
* Connection #0 to host localhost left intact
* Closing connection 0

@belyalov I use the latest docker image of envoy-dev. I will verify that again.

@belyalov I put an example here to reproduce it (https://github.com/npuichigo/grpc_with_envoy). Here I re-pull the latest docker image of envoy-dev.
I tried to set content-type once and it works well. Multiple chunked responses even with the same content-type make the received content-type application/json, but I checked the received data and it's right without wrapping by json.
$ curl "localhost:8080/get?filename=testdata/small.mp3" -4 -v > test_chunk_music.mp3
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying <IP_ADDRESS>...
* Connected to localhost (<IP_ADDRESS>) port 8080 (#0)
> GET /get?filename=testdata/small.mp3 HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: audio/mp3
< grpc-accept-encoding: identity,deflate,gzip
< accept-encoding: identity,gzip
< x-envoy-upstream-service-time: 0
< date: Fri, 24 Apr 2020 03:26:57 GMT
< server: envoy
< transfer-encoding: chunked
<

$ curl "localhost:8080/get?filename=testdata/music.mp3" -4 -v > test_chunk_music.mp3
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying <IP_ADDRESS>...
* Connected to localhost (<IP_ADDRESS>) port 8080 (#0)
> GET /get?filename=testdata/music.mp3 HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json
< grpc-accept-encoding: identity,deflate,gzip
< accept-encoding: identity,gzip
< x-envoy-upstream-service-time: 0
< date: Fri, 24 Apr 2020 03:27:24 GMT
< server: envoy
< transfer-encoding: chunked
<

@belyalov I put an example here to reproduce it (https://github.com/npuichigo/grpc_with_envoy). Here I re-pull the latest docker image of envoy-dev.
I found that it's cause by the size of HttpBody response. If the body size is smaller than 16KB, then it works well. Otherwise, the received content type is application/json. So is it by design and is there any setting to configure it?
You can change static const int kBufferSize = 64 * 1024; in https://github.com/npuichigo/grpc_with_envoy/blob/master/grpc_with_envoy/serving/demo_service_impl.cc to reproduce that.

I see the problem is in this line
https://github.com/envoyproxy/envoy/blob/master/source/extensions/filters/http/grpc_json_transcoder/json_transcoder_filter.cc#L526
If buildResponseFromHttpBodyOutput() did not receive a full HttpBody data frame, it should not return "continue",  it should return Stop to keep buffering the data.

@belyalov

@qiwzhang So do you think buildResponsFromHttpBodyOutput should return a boolean to indicate whether we receive a full data frame? Or do we need to check that only for HttpBody?
if (frames.empty()) {
    return true;
}

@npuichigo and @qiwzhang
Sorry guys, I missed this thread.
Got the repro and @qiannawang is right - basically what happens where is when Http::FilterHeadersStatus::Continue returned - envoy sends headers, that's why you see application/json as content type.
Going to fix it very soon, thanks for research!


Implement bundle budgets functionality in esbuild-based browser application builder

Command
build
Description
Implement the budgets option and related functionality for the esbuild-based browser application builder.
Describe the solution you'd like
No response
Describe alternatives you've considered
No response

This has been implemented and is available in 17.0.0-next.8


Bug/fix broken constructor

This fixes https://github.com/mailosaur/mailosaur-dotnet/issues/10.
Tests are failing on master for me (it looks like some dog image asset changed?), and continue to fail on this branch:
Test run for C:\code\mailosaur-dotnet\Mailosaur.Test\bin\Debug\netcoreapp2.0\Mailosaur.Test.dll(.NETCoreApp,Version=v2.0)
Microsoft (R) Test Execution Command Line Tool Version 15.9.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
[xUnit.net 00:00:21.3200694]     Mailosaur.Test.EmailsTests.WaitForTest [FAIL]
Failed   Mailosaur.Test.EmailsTests.WaitForTest
Error Message:
 Assert.Equal() Failure
           (pos 0)
Expected: Resources/dog.png
Actual:   dog.png
           (pos 0)
Stack Trace:
   at Mailosaur.Test.EmailsTests.ValidateAttachmentMetadata(Message email) in C:\code\mailosaur-dotnet\Mailosaur.Test\EmailsTests.cs:line 280
   at Mailosaur.Test.EmailsTests.ValidateEmail(Message email) in C:\code\mailosaur-dotnet\Mailosaur.Test\EmailsTests.cs:line 182
   at Mailosaur.Test.EmailsTests.WaitForTest() in C:\code\mailosaur-dotnet\Mailosaur.Test\EmailsTests.cs:line 83
[xUnit.net 00:00:22.3378639]     Mailosaur.Test.EmailsTests.GetTest [FAIL]
Failed   Mailosaur.Test.EmailsTests.GetTest
Error Message:
 Assert.Equal() Failure
           (pos 0)
Expected: Resources/dog.png
Actual:   dog.png
           (pos 0)
Stack Trace:
   at Mailosaur.Test.EmailsTests.ValidateAttachmentMetadata(Message email) in C:\code\mailosaur-dotnet\Mailosaur.Test\EmailsTests.cs:line 280
   at Mailosaur.Test.EmailsTests.ValidateEmail(Message email) in C:\code\mailosaur-dotnet\Mailosaur.Test\EmailsTests.cs:line 182
   at Mailosaur.Test.EmailsTests.GetTest() in C:\code\mailosaur-dotnet\Mailosaur.Test\EmailsTests.cs:line 57

Total tests: 16. Passed: 14. Failed: 2. Skipped: 0.
Test Run Failed.
Test execution time: 33.2519 Seconds

We cannot reproduce the test failure - we also regularly run tests via Travis and don't have failures. Could be the version of dotnet you're running?

#10 has been resolved via https://github.com/mailosaur/mailosaur-dotnet/commit/8153c5dcd856e9968287bc1a73f33061e164f8be


CSS imports

Yellow Lab Tools reports several uses of @import which should be avoided to significantly improve its global score.

What way would you seperate the files? Is your suggestion to combine them into one? Is that making it better?

Yellow Lab Tools suggests to use <link rel='stylesheet' href='a.css'> instead.


All custom events dissapeared

Hi, all our custom events has suddenly disappeared from the calendar, everything worked fine yesterday but seems like the latest release contains a bug. Could you please fix or revert the latest release?

@afernandez-pc , can you share which version of the extension you have (2.0.22?) and which Azure DevOps version you are on?

Thanks. We are collecting feedback, so far no new issue reported on 2020. Is this the same as #442.

@MOlausson There might be a different issue more severe than #442 where the events reappear after navigation. Our events are missing and never reappear even after navigating. We're on 2.0.22.

Note: newly created events show up as expected. The dozens of events created prior to today are nowhere to be found...

Same here: all our planned dates are gone since Monday (v 2.0.21).  Newly crated dates will show up.
Please fix or revert. v2.0.22 is no improvement so far. Maybe the error exists since 2.0.18 - this I can't tell.

Are there any updates on this?  If we don't have this resolved, I think we have to abandon using this extension and come up with a new tool, thanks!

@MOlausson  We have the same issue. lost all the custom events.
Team Calendar is updated to version 2.0.22
Version - Azure DevOps Server 2020 Update 1.1
Note: newly created events show up as expected. The dozens of events created prior to today are nowhere to be found...

We are still investigating this issue. Would anyone be able to confirm if events created prior to Feb 23 are shown?

No, not even events created prior to Feb 23, they're all gone. :(
Only newly-created ones are shown now.

We are still investigating this issue. Would anyone be able to confirm if events created prior to Feb 23 are shown?

No, not showing any events created prior to 9th of March, only showing the one created after 9th of March.

We don't see any events at all regardless of date and if we try to add new events, we get a security popup that we can't get past.

We regret to inform you that we could not recover the data you reported. If you have any questions, please feel free to contact us, and we will do our best to assist you.


Docs/improve readme

Fixes Issue
closes #52

Everything looks acceptable to me but I think we should use some emojis to make the readme more enjoyable as I see many big companies doc are using emojis. what do you have to suggest on that?

Hey @rohitdasu , One more thing the demo should be after the project description. Could you check that!
__
  Summary: :question: I have a question

@RakeshSangem what do you think about icons?

@RakeshSangem what do you think about icons?

I believe we can use, but not excessively.
__
  Summary: :hammer_and_wrench: This code needs a fix

@RakeshSangem what do you think about icons?

I believe we can use, but not excessively.
__        Summary: 🛠 This code needs a fix

imo we should keep the docs as professional as possible.

@rohitdasu   imo we should keep the docs as professional as possible.

In that case we can skip them.
__
  Tags: Brittle


15 use search opts in android

depends on #5

depends on #5

Do you mean it is branched off of #5 ? If so can we point the review at #5's branch rather than master please?


Sharing with expiration date contract test against oC10 server failed

Build: https://drone.owncloud.com/owncloud/owncloud-sdk/4211/9/11
oC10 server: 10.12.0 stable 2023-03-13
as 'testUser1', a POST request to share a file 'test space and + and #.txt' to user with expiration date set
    returns a response which
      has status code 200 (OK)
      includes headers
        "Content-Type" with value "text/xml; charset=utf-8" (OK)
      has a matching body (FAILED)
  as 'testUser1', a POST request to share a file 'test.txt' to user with expiration date set
    returns a response which
      has status code 200 (OK)
      includes headers
        "Content-Type" with value "text/xml; charset=utf-8" (OK)
      has a matching body (FAILED)
  as 'testUser1', a POST request to share a file '文件.txt' to user with expiration date set
    returns a response which
      has status code 200 (OK)
      includes headers
        "Content-Type" with value "text/xml; charset=utf-8" (OK)
      has a matching body (FAILED)

Failures:

1) Verifying a pact between owncloud-sdk and oc-server-pendingOn-ocis Given the user is recreated And the user is recreated And file exists - as 'testUser1', a POST request to share a file 'test space and + and #.txt' to user with expiration date set returns a response which
    1.1) has a matching body
           $.ocs.meta.message.#text -> Expected 'Expiration date is in the past' to match '(OK)?'
           $.ocs.meta.statuscode.#text -> Expected '100' to be equal to '404'
           $.ocs.data -> Expected child <displayname_file_owner/> but was missing
           $.ocs.data -> Expected child <displayname_owner/> but was missing
           $.ocs.data -> Expected child <expiration/> but was missing
           $.ocs.data -> Expected child <file_target/> but was missing
           $.ocs.data -> Expected child <id/> but was missing
           $.ocs.data -> Expected child <path/> but was missing
           $.ocs.data -> Expected child <permissions/> but was missing
           $.ocs.data -> Expected child <share_type/> but was missing
           $.ocs.data -> Expected child <stime/> but was missing
           $.ocs.data -> Expected child <uid_file_owner/> but was missing
           $.ocs.data -> Expected child <uid_owner/> but was missing
           $.ocs.meta.status.#text -> Expected 'ok' to be equal to 'failure'
2) Verifying a pact between owncloud-sdk and oc-server-pendingOn-ocis Given the user is recreated And the user is recreated And file exists - as 'testUser1', a POST request to share a file 'test.txt' to user with expiration date set returns a response which
    2.1) has a matching body
           $.ocs.meta.message.#text -> Expected 'Expiration date is in the past' to match '(OK)?'
           $.ocs.meta.statuscode.#text -> Expected '100' to be equal to '404'
           $.ocs.data -> Expected child <displayname_file_owner/> but was missing
           $.ocs.data -> Expected child <displayname_owner/> but was missing
           $.ocs.data -> Expected child <expiration/> but was missing
           $.ocs.data -> Expected child <file_target/> but was missing
           $.ocs.data -> Expected child <id/> but was missing
           $.ocs.data -> Expected child <path/> but was missing
           $.ocs.data -> Expected child <permissions/> but was missing
           $.ocs.data -> Expected child <share_type/> but was missing
           $.ocs.data -> Expected child <stime/> but was missing
           $.ocs.data -> Expected child <uid_file_owner/> but was missing
           $.ocs.data -> Expected child <uid_owner/> but was missing
           $.ocs.meta.status.#text -> Expected 'ok' to be equal to 'failure'
3) Verifying a pact between owncloud-sdk and oc-server-pendingOn-ocis Given the user is recreated And the user is recreated And file exists - as 'testUser1', a POST request to share a file '文件.txt' to user with expiration date set returns a response which
    3.1) has a matching body
           $.ocs.meta.statuscode.#text -> Expected '100' to be equal to '404'
           $.ocs.data -> Expected child <displayname_file_owner/> but was missing
           $.ocs.data -> Expected child <displayname_owner/> but was missing
           $.ocs.data -> Expected child <expiration/> but was missing
           $.ocs.data -> Expected child <file_target/> but was missing
           $.ocs.data -> Expected child <id/> but was missing
           $.ocs.data -> Expected child <path/> but was missing
           $.ocs.data -> Expected child <permissions/> but was missing
           $.ocs.data -> Expected child <share_type/> but was missing
           $.ocs.data -> Expected child <stime/> but was missing
           $.ocs.data -> Expected child <uid_file_owner/> but was missing
           $.ocs.data -> Expected child <uid_owner/> but was missing
           $.ocs.meta.message.#text -> Expected 'Expiration date is in the past' to match '(OK)?'
           $.ocs.meta.status.#text -> Expected 'ok' to be equal to 'failure'

There were 3 pact failures

The nightly CI is pulling the wrong contract version.

The nightly CI is pulling the wrong contract version.

true. Weirdly was pulling version created by https://github.com/owncloud/owncloud-sdk/pull/1192/commits/458f5a1c56d2ca65fbadf71ed82d04991e0e1b96
Hopefully after merge of this PR https://github.com/owncloud/owncloud-sdk/pull/1227, new version will be created

Nightly is passing after the above PR being merged. So closing this issue.


Add support to read ssl connection params from the config file

Support to read ssl connection params from the config file
Description
I didn't add the ssl conection params support to config file feature, so I did in this PR.
Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] This change requires a documentation update

How Has This Been Tested?
I ran the test suite.
Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] I have checked my code and corrected any misspellings

Looks good to me!


Colorscheme non-systemed list above systemed.

And they will list upper.

blue, darkblue, default list in bottom.


Remove unnecessary debug print from hamming distance calculation

Fixes #27

@marinegor if you update the version in cargo.toml then we can do a release immediately


core IBC does not support timeouts for solo machine

Summary
The integration between core IBC <> solo machine is incapable of processing timeouts.
Problem Definition
The solo machine does not set consensus states for every height it signs a sequence, it only maintains a latest timestamp and has no sense of past timestamps.
This is accounted for in SendPacket, but not in TimeoutPacket
Decision
Apply the same fix for pre-v7 versions onto the function call in TimeoutPacket. Since the issue was reported via internal auditing, I don't think this fix warrants a minor release to fix logic by itself (let the fix be backported and naturally released with other changes). This issue does not affect main
Thanks @AdityaSripal for reporting the issue

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate contributors tagged/assigned

Closing the issue. I believe this issue wasn't previously fixed because it was known timeouts didn't work which would require adding solo machine specific query logic into 04-channel. The fix in send packet is ignored for solo machines because it is defensive, while the call in timeout is required


run_s2vt.py:62: SettingWithCopyWarning:  A value is trying to be set on a copy of a slice from a DataFrame.

Hi, when I ran the code for testing, it gave me some error like
""
run_s2vt.py:62: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
train_vids['video_path'] = train_vids['video_id'].map(lambda x: os.path.join(cfg.path_to_trainval_descriptors, x + "_incp_v3.npy"))
run_s2vt.py:63: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
val_vids['video_path'] = val_vids['video_id'].map(lambda x: os.path.join(cfg.path_to_trainval_descriptors, x + "_incp_v3.npy"))
{'CIDEr': 0.38908523814484475, 'Bleu_4': 0.3488993782941496, 'Bleu_3': 0.4645981761823224, 'Bleu_2': 0.6031291787236476, 'Bleu_1': 0.7600115088383854, 'ROUGE_L': 0.57838682993085411, 'METEOR': 0.26565351293713196}
2990
""
Do you know what is going on the testing code? Appreciate any kinds help.

This is just a warning, not an error, do not care about it

That's what I thought so but the program stopped after this warning tho

@xingtengfei, thanks.
Indeed, it shouldn't give problems in this case. The rest of the output seems correct, isn't it? I'll take care of it later.

@ramanishka Hi,  I got this warning as well when i ran the code for testing  using python3.6 on win10. Do you know how to get rid of this warning? Thanks~


bin/render_notebooks.sh installs sympy master

This causes CI to fail on the cythonizing notebook which now makes use of the new autowrap features in 1.1. I think that line can just be removed?

I think you need to replace the line with the 1.1rc tar ball link, no?

Changed with 09ae029, though shouldn't it work without this since sympy is installed in the conda environment that's sourced right before?

Oh yeah, I guess so.

CircleCI caches the conda environment. I rebuilt without cache. This is now failing:
# get a path to the cfib source
import os
import scipy2017codegen
cfib_path = os.path.join(os.path.dirname(scipy2017codegen.__file__), 'cfib')

# generate a "pyxbld" file that specifies how to build everything
from scipy2017codegen.templates import render_pyxbld
render_pyxbld('fib',
              sources=[os.path.join(cfib_path, 'cfib.c')],
              include_dirs=[np.get_include(), cfib_path])
[NbConvertApp] output: status
[NbConvertApp] output: execute_input
[NbConvertApp] output: status
[NbConvertApp] Executing cell:
%%cython_pyximport fib
import numpy as np
cimport numpy as np

# here we just replicate the function signature from the header
cdef extern from "cfib.h":
    void cfib(int n, double *x)

# here is the "wrapper"
def fib(int n):
    # preallocate an array of doubles
    cdef np.ndarray[np.float64_t, ndim=1, mode='c'] x
    x = np.empty(n, dtype=np.float64)

    # call the c function and return the result
    cfib(n, <double *> x.data)
    return x
In file included from /home/ubuntu/miniconda/envs/codegen17/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1788:0,
                 from /home/ubuntu/miniconda/envs/codegen17/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /home/ubuntu/miniconda/envs/codegen17/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from /home/ubuntu/.pyxbld/temp.linux-x86_64-3.6/pyrex/fib.c:435:
/home/ubuntu/miniconda/envs/codegen17/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^
/home/ubuntu/.pyxbld/temp.linux-x86_64-3.6/pyrex/fib.c:437:18: fatal error: cfib.h: No such file or directory
 #include "cfib.h"
                  ^
compilation terminated.
[NbConvertApp] output: status
[NbConvertApp] output: execute_input
[NbConvertApp] output: error
[NbConvertApp] output: status
[NbConvertApp] ERROR | Error while converting 'notebooks/08-cythonizing.ipynb'
Traceback (most recent call last):
  File "/home/ubuntu/miniconda/envs/codegen17/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 381, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/home/ubuntu/miniconda/envs/codegen17/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 172, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/home/ubuntu/miniconda/envs/codegen17/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 190, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/home/ubuntu/miniconda/envs/codegen17/lib/python3.6/site-packages/nbconvert/exporters/html.py", line 84, in from_notebook_node
    return super(HTMLExporter, self).from_notebook_node(nb, resources, **kw)
  File "/home/ubuntu/miniconda/envs/codegen17/lib/python3.6/site-packages/nbconvert/exporters/templateexporter.py", line 268, in from_notebook_node
    nb_copy, resources = super(TemplateExporter, self).from_notebook_node(nb, resources, **kw)
  File "/home/ubuntu/miniconda/envs/codegen17/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 132, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/home/ubuntu/miniconda/envs/codegen17/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 309, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/home/ubuntu/miniconda/envs/codegen17/lib/python3.6/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb,resources)
  File "/home/ubuntu/miniconda/envs/codegen17/lib/python3.6/site-packages/nbconvert/preprocessors/execute.py", line 242, in preprocess
    nb, resources = super(ExecutePreprocessor, self).preprocess(nb, resources)
  File "/home/ubuntu/miniconda/envs/codegen17/lib/python3.6/site-packages/nbconvert/preprocessors/base.py", line 70, in preprocess
    nb.cells[index], resources = self.preprocess_cell(cell, resources, index)
  File "/home/ubuntu/miniconda/envs/codegen17/lib/python3.6/site-packages/nbconvert/preprocessors/execute.py", line 275, in preprocess_cell
    raise CellExecutionError(msg)
nbconvert.preprocessors.execute.CellExecutionError: An error occurred while executing the following cell:
------------------
%%cython_pyximport fib
import numpy as np
cimport numpy as np

# here we just replicate the function signature from the header
cdef extern from "cfib.h":
    void cfib(int n, double *x)

# here is the "wrapper"
def fib(int n):
    # preallocate an array of doubles
    cdef np.ndarray[np.float64_t, ndim=1, mode='c'] x
    x = np.empty(n, dtype=np.float64)

    # call the c function and return the result
    cfib(n, <double *> x.data)
    return x
------------------

ImportError: Building module fib failed: ["distutils.errors.CompileError: command 'gcc' failed with exit status 1\n"]

[NbConvertApp] Exiting application: jupyter-nbconvert

./bin/render_notebooks.sh returned exit code 1

CircleCI builds with environment-gcc.yml, I'm surprised it makes a difference here.

@bjodah Hopefully I just fixed that. I hadn't added the cfib/cfib.{c,h} files to MANIFEST.in so they weren't getting packaged.


Enable E2E tests in CI pipeline

Some crates have E2E tests in addition to local ones. These tests ensure the correctness of the implemented APIs by checking them against the Azure platform.
We have to decide if it makes sense to reenable these tests in the current CI pipeline.
PS: We can kick off the tests with cargo test --features=test_e2e in the supported modules.

I don't think e2e tests make sense since most of them require an active Azure subscription to run. We are now checking that the tests actually compile. Perhaps we can consider a bors style PR merger at some point, that allows us to kick off e2e tests manually through the GitHub PR interface, but I think it would be a very bad idea to run these tests automatically.

Yes it's a good idea.
The problems are that the E2E tests are, now, neither idempotent nor resilient.
That means that a failed test leaves the subscription in an undetermined state possibly causing subsequent tests to fail.
Also, sometimes Azure services return a transient error that could make the test fail even if they should not.
Until those two problems are fixed I believe your approach is the best. 👍