Fix printing of complex numbers

Because complex numbers print with either a + or - operator, the expression printing can have expressions $x * c$, for $c$ a constant, which actually gets printed like $x * 1.0 - 2.0im$, which looks like $x * c - b$ for two constants.
This PR implements a fix - any number which is neither a real number nor a vector will be printed inside an additional parentheses.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

6 of 6 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 90.909%

Totals

Change from base Build<PHONE_NUMBER>:
0.2%

Covered Lines:
950

Relevant Lines:
1045

💛  - Coveralls


Optionally skip reading out superscripts.

Reading out superscript numbers as "footnote 1," "footnote 2," etc can be helpful, but likely in very limited cases. It would be nice if there was either an option to toggle superscript reading off specifically, or overall a verbosity setting which could go from how the app reads now (most verbose) to where the app only reads words and lead numbers (least verbose). The superscripts could even be replaced with a ping sound in background, or similar, which acknowledges the presence of a note, without interrupting the reading.
I've found sometimes stylizing things like first as "1st" will make the app read "one-S.-T." or similar. In those cases, the flow of the reading could be less interrupted even if the app just read it as "One" and ignored the tail characters after the one. This one might require further investigation, but the superscripts are definitely skippable/replaceable.

Sorry, I won't be doing this as I'm busy working on other projects now. I myself am using https://github.com/lesleyrs/tts-tui right now (also unmaintained but it works well on Windows). When I let tts-tui read 1st it will pronounce it correctly as it uses the newer winrt speech api.
About your first problem, I just try to avoid copying too much at once and it hasn't been much of an issue.


Removed tests out of graph mode in weight decay optimizer test

cc @gabrieldemarmiesse

@philjd
You are owner of some files modified in this pull request.
Would you kindly review the changes whenever you have the time to?
Thank you very much.

Looks great, thanks!


Remove flicker on relaunch

Summary of the Pull Request
What is this about?
PR Checklist

[ ] Applies to #xxx
[ ] CLA signed. If not, go over here and sign the CLA
[ ] Tests added/passed
[ ] Requires documentation to be updated
[ ] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

Info on Pull Request
What does this include?
Validation Steps Performed
How does someone test & validate?

awesome you got it working!


Fix CVE-2023-3817 Excessive time spent checking DH q parameter value [1.1.1]

Fix excessive time spent checking DH q parameter value.
The function DH_check() performs various checks on DH parameters. After
fixing CVE-2023-3446 it was discovered that a large q parameter value can
also trigger an overly long computation during some of these checks.
A correct q value, if present, cannot be larger than the modulus p
parameter, thus it is unnecessary to perform these checks if q is larger
than p.
CVE-2023-3817

CI failure is not relevant.

24 hours has passed since 'approval: done' was set, but as this PR has been updated in that time the label 'approval: ready to merge' is not being automatically set.  Please review the updates and set the label manually.

Pushed. Thanks.


Slow download the documents

Hi benyap,
Firstly, I would like to thank you for version 2.x, it is so better, in the 1.x I was having some problems connecting on the Firebase Emulator, sometimes it connected but sometimes did not, but now it is working fine.
So, now I am finding it is slow to download the documents, when I am using the Firebase Emulator, it downloads about 10 documents and when using the production, it downloads about 40 documents per 5 seconds, I do not know if this slow is because of download document or when saving in the JSON. Do you think there is a way to do it faster? The reason for my question is because my data it big.
It is part of the logs:

export info    Export data from demo-backup to backfile.ndjson
export verbose {
options: {
paths: [ 'xxxx/xxxx/xxxx/xxxx' ],
exploreInterval: 10,
exploreChunkSize: 5000,
downloadChunkSize: 5000,
downloadInterval: 10,
overwrite: true,
debug: true,
verbose: true
},
connection: { project: 'demo-backup', emulator: 'localhost:8180' }
}
export debug   Overwriting existing data at backfile.ndjson
export debug   Progress: 0 exported, 0 exploring, 1 to explore
export verbose Exploring for subcollections in 1 document
export verbose Exporting 1 document
export debug   Progress: 0 exported, 1 exploring, 0 to explore
export verbose Found 3 subcollections in xxxx/xxxx/xxxx/xxxx
export verbose Exploring for documents in 3 collections
export verbose Found 1 document in xxxx/xxxx/xxxx/xxxx/apps
export verbose Exporting 1 document
export debug   Progress: 2 exported, 2 exploring, 1 to explore
export verbose Found 10163 documents in xxxx/xxxx/xxxx/xxxx/xxxx
export verbose Exporting 5000 documents
export verbose Found 24 documents in xxxx/xxxx/xxxx/xxxx/xxxx
export verbose Exploring for subcollections in 5000 documents
export verbose Exporting 5000 documents
export debug   Progress: 5002 exported, 5000 exploring, 5188 to explore
export verbose Found 2 subcollections in xxxx/xxxx/xxxx/xxxx/xxxx/xxxx
export verbose Exporting 187 documents
export debug   Progress: 10189 exported, 4990 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4980 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4975 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4966 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4955 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4953 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4942 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4932 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4929 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4918 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4909 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4905 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4894 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4885 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4883 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4873 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4864 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4861 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4851 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4841 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4839 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4831 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4821 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4815 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4809 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4799 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4792 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4788 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4778 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4769 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4766 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4756 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4747 exploring, 5190 to explore
export debug   Progress: 10189 exported, 4744 exploring, 5190 to explore

Hi @anderjf, thanks for using Firestore Backfire and I'm glad it's working better!
A few questions:

To clarify, are you saying that it is faster when exporting from the emulator or from a production Firestore instance?
I noticed that you've changed some of the advanced config options (exploreInterval, exploreChunkSize, downloadInterval, downloadChunkSize). What was the behaviour like with the defaults compared to what you set?
When you say that your data is big, do you mean that each individual document is large? Or that you have a lot of records? Any estimates on how big we're talking here? (average document size, number of documents etc.)

In my experience, the slowest part of exporting data from Firestore are:

Listing a large number of documents from a collection using listDocuments()
Downloading a large number of documents using getAll(...)

Unfortunately if you have a collection with a large number of documents, it's just going to be slow given the nature of how listDocuments is implemented. I might try using the REST API and paginating the calls myself at some point to see if that helps speed things up when I have the time.
Looking at your logs, it doesn't seem like the bottleneck is downloading and saving the JSON data. It managed to export over 10k documents, then it kinda sits around exploring paths without exporting anything.
Assuming that the "Progress" logs are being printed every 5 seconds (which is the default behaviour), I'm a little unsure as to why it's exploring so slowly. It is calling listDocuments() for each document it's exploring, but the fact that it hasn't reported it having found any subcollections means that the call itself should have returned very quickly, and it should easily get through all of them within a few seconds.
If you don't mind, could you share what the structure of your data is? It will be helpful to understand how deeply nested your data is, and how many subcollections/documents there are.

HI @benyap,
Answering your questions:

From a production Firestore instance is faster, but not so much.
The same results.
I was meaning that my database has a lot of records, but the only thing like size, it would be the backup files that I exported from production, which have about 652MB.

Maybe you could also try to use stream, I realized that faster than listDocuments:
  const account = db.collection(path);

  let count = 0;

  account.stream().on('data', (documentSnapshot) => {
    console.log(documentSnapshot.ref.path);
    ++count;
  }).on('end', () => {
    console.log(`Total count is ${count}`);
  });

I don't know if it will help, but it is more or less what we have.
{
  "col": {
    "doc": {
      "__collections__": {
        "col": {
          "doc": {
            "__collections__": {
              "col": {
                "doc": {
                  "__collections__": {
                    "col": {  /** in this collection we have about 11k documents **/
                      "doc": {
                        "field": "value",
                        "field": "value"
                      },
                      "doc": {
                        "field": "value",
                        "field": "value"
                      },
                      "doc": {
                        "field": "value",
                        "field": "value"
                      }
                    },
                    "col": { /** in this collection we have about 200k documents **/
                      "doc": {
                        "field": "value",
                        "field": "value"
                      },
                      "doc": {
                        "field": "value",
                        "field": "value"
                      },
                      "doc": {
                        "field": "value",
                        "field": "value"
                      }
                    }
                  },
                  "field": "value",
                  "field": "value"
                }
              }
            },
            "field": "value",
            "field": "value"
          }
        }
      }
    }
  }
}

Thank you for your help.

Hey @anderjf,
I did some digging and tried using stream() as you suggested, and it indeed was faster as it allows concurrent processing of data as it streams in. This change has been released in v2.1.0, so please give it a shot and let me know how it goes!
Closing this for now. Feel free to reopen if you run into any issues related to this.

Hi @benyap,
I'm glad to have helped.
So, I tested it, and it is working well and downloading faster, but the bottleneck now is in exploring if the document has any collection, see the logs.
export 8:13:32 am info    Export data from demo-backup to backfile.ndjson
export 8:13:32 am verbose {
  options: {
    paths: [ 'xxxx/xxxx/xxxx/xxxx/risksxxxx' ],
    overwrite: true,
    debug: true,
    verbose: true
  },
  connection: { project: 'demo-backup', emulator: 'localhost:8180' }
}
export 8:13:32 am debug   Overwriting existing data at backfile.ndjson
export 8:13:32 am debug   Progress: 0 exported, 1 to export, 0 exploring, 1 to explore, 0 reads
export 8:13:32 am debug   Exploring for documents in 1 collection
export 8:13:33 am verbose Exporting 9736 documents...
export 8:13:33 am debug   Successfully exported 9736 documents
export 8:13:33 am debug   Exploring for subcollections in 1000 documents
export 8:13:34 am verbose Exporting 427 documents...
export 8:13:34 am debug   Successfully exported 427 documents
export 8:13:37 am debug   Progress: 10163 exported, 0 to export, 991 exploring, 9163 to explore, 10163 reads
export 8:13:42 am debug   Progress: 10163 exported, 0 to export, 979 exploring, 9163 to explore, 10163 reads
export 8:13:47 am debug   Progress: 10163 exported, 0 to export, 966 exploring, 9163 to explore, 10163 reads
export 8:13:52 am debug   Progress: 10163 exported, 0 to export, 954 exploring, 9163 to explore, 10163 reads
export 8:13:57 am debug   Progress: 10163 exported, 0 to export, 942 exploring, 9163 to explore, 10163 reads
export 8:14:02 am debug   Progress: 10163 exported, 0 to export, 930 exploring, 9163 to explore, 10163 reads
export 8:14:07 am debug   Progress: 10163 exported, 0 to export, 917 exploring, 9163 to explore, 10163 reads
export 8:14:12 am debug   Progress: 10163 exported, 0 to export, 905 exploring, 9163 to explore, 10163 reads
export 8:14:17 am debug   Progress: 10163 exported, 0 to export, 893 exploring, 9163 to explore, 10163 reads
export 8:14:22 am debug   Progress: 10163 exported, 0 to export, 881 exploring, 9163 to explore, 10163 reads
export 8:14:27 am debug   Progress: 10163 exported, 0 to export, 869 exploring, 9163 to explore, 10163 reads
export 8:14:32 am debug   Progress: 10163 exported, 0 to export, 857 exploring, 9163 to explore, 10163 reads
export 8:14:37 am debug   Progress: 10163 exported, 0 to export, 845 exploring, 9163 to explore, 10163 reads
export 8:14:42 am debug   Progress: 10163 exported, 0 to export, 832 exploring, 9163 to explore, 10163 reads
export 8:14:47 am debug   Progress: 10163 exported, 0 to export, 820 exploring, 9163 to explore, 10163 reads
export 8:14:52 am debug   Progress: 10163 exported, 0 to export, 808 exploring, 9163 to explore, 10163 reads
export 8:14:57 am debug   Progress: 10163 exported, 0 to export, 796 exploring, 9163 to explore, 10163 reads
export 8:15:02 am debug   Progress: 10163 exported, 0 to export, 784 exploring, 9163 to explore, 10163 reads

In this case, there are 10163 documents, and it will check for each document if there are some subcollection, I don't know if has some way to do that faster, maybe to change this verification to be a parallel verification, using a Promise.

Thanks for the feedback @anderjf. Will look into this soon 😊

Hi @anderjf, an enhancement to improve the subcollection exploration speed has been released in v2.2.0 😊


added profile location for NPL

after fiddling with transaction RZ10 (which eded up in breaking my original DEFAULT.PFL file) - I realised that it is much simpler to edit DEFAULT.PFL directly. Maybe this information could helps others?
( my miniSAP is NPL 7.51 / Sybase / openSUSE )

thank you for info on A4H - I have added it to the list.
regarding providing official titles - I've been looking for a better way of specifying appropriate product title and decided to go with same titles/classification as listed on the License Keys page. From that page I understand that for evaluation versions (i.e. miniSAP) there is a unique predefined system-name for particular combination of NetWeaver / database / operating system.

( plz let me know if I can improve this further )

Great. If you like add the link to the license key page to the docs.

add the link to the license key page

good idea, thank you .. also sorted the list by system-name alphabetically.

thanks @alex-salnikov
welcome to abapGit as a contributor 🎉


Fix prompt only when changes made

PR Description
Only shows save prompt when changes are made to the requests
Related Issues

Related Issue #364
Closes #364

Checklist

[x] I have gone through the contributing guide
[x] I have updated my branch and synced it with project main branch before making this PR
[x] I have run the tests (flutter test) and all tests are passing

Added/updated tests?

[x] No, and this is why: Not Required

Idea is good. Approach is incorrect. You need to think more to figure out a better way.
Refer to the answer written by the author of riverpod here - https://stackoverflow.com/questions/74466072/riverpod-ref-watchstatenotifierprovider-vs-ref-watchstatenotifierprovider-not

Modify changeMade -> hasUnsavedChanges everywhere.

Thanks for the PR. LGTM 🚀


Added Krita and Just Color Picker, rearranged some entries

Krita is a painting and graphic design software, Just color picker is a portable color picker and editor.

oh, how I love resolving conflicts in branches :)

@andersonaddo thanks for contributing good tools and putting the right labels ❤


Feature: Audio seeking bar

Hi @ricardopereira , I assume u already have audio seeking bar in mind... I want to implement this, can u give me some insights on what u already have in mind?

@ricardopereira my next app will have an audio seeking bar, wondering what idea do you have to implement this? (I see you commented some test relating audio seeking)


Add HLL++ support for better accuracy and possibly lower memory cost

DISTINCTCOUNTHLL accuracy and memory footprint can be improved through latest HLL algorithms. We have a choice either replace the existing implementation with a better one or leave the existing DISTINCTCOUNTHLL to implement original HLL and create separate functions (ex: DISTINCTCOUNTHLLPLUSPLUS).
Google's HLL++ -- a popular algorithm amongst the community offers lot of improvements over original HLL. There are multiple java implementations of HLL++. Most of them have variations in performance due to the register size and other implementation choices. Clearspring stream-lib used for current HyperLogLog function, implements HLL++ as HyperLogPlus.

Another alternative is the Data Sketch HLL: https://datasketches.apache.org/docs/HLL/HLL.html
They claim better performance in this paper: https://datasketches.apache.org/docs/HLL/Hll_vs_CS_Hllpp.html

Good point @Jackie-Jiang. We could add this as a separate function DISTINCTCOUNTHLLSKETCH (similar to DISTINCTCOUNTTHETASKETCH). It would be nice to document the parameters, accuracy and space requirements of each of HLL implementations as we have a handful of them already. Users can chose the right one based on their need.


A question about the augmentation method

HI, the repo is a nice work, thanks for your sharing.
I want to know if these augmentation methods are effective,
like the RandomErasing/Mixup/RandAugment/Cutout/CutMix?

Thanks for your interset. I have not yet compared the effects of these data augmentation methods, i will conduct a series of contrast experiment soon.

@cavalleria thanks for your reply. Looking forward to the results of your experiments.

@cavalleria thanks for your reply. Looking forward to the results of your experiments.

@Gary-Deeplearning the data augmentation result is update in model_zoo.

I think for the data augmentation, you need to train it longer.
You can also resume from baseline and train from there instead (which is what I do). I always seem to get more accuracy than baseline.

yes , data augmentation may need to train more epochs. can you provide your experiment result and training details?

@xsacha  if it's possiable, you can add results that resume training in model_zoo

@cavalleria I checked the result from model_zoo. It seems the data aug get lower accuracy?

@cavalleria I checked the result from model_zoo. It seems the data aug get lower accuracy?

The results that i trained from scratch have the same hyperparameters with baseline seem not better than baseline.

@cavalleria my results wouldn't be compatible as I use a different face detector and dataset(s).
I start with the fully trained baseline backbone and retrain with LR=0.01

@cavalleria my results wouldn't be compatible as I use a different face detector and dataset(s).
I start with the fully trained baseline backbone and retrain with LR=0.01 + augmentations turned on
Some blur augmentations are good btw. I use a third-party package (kornia) to do Motion Blur (very effective!) and Gaussian Blur. I also have success improving accuracy with the augmentations in here.
I think the comparisons done in Model Zoo probably aren't very helpful though as augmented data is always going to perform worse than non-augmented with the same training time. It's more information for the model to learn.

In my experience, blur augmentation is definitely effective. I think comparision with same hypeparameter is more fair, but if you want to imporve accuracy , you can tune hypeparameters and finetune models.

The purpose of cutmix, mixup is to increase local information, and can also classify objects when they are occluded. it is usefull in object detection. There is little occlusion information in face recognition data. if we change the cutmix(cut the training data from the middle of the face), Side face effect may be improved. If anyone has time, It's worth trying

I've tried this kind of augmentation by erasing half of the aligned face to simulate face with mask. Loss decreased normally while accuracy was worse. My iteration was the same as baseline which may be the reson for this situation.

I've tried this kind of augmentation by erasing half of the aligned face to simulate face with mask

the cutmix random crop(crop size is random,not fixed) from one batch size images and generate a new pic and target. I mean divide the face into two parts according to the left and right. if you use cutmix. change the codes and let different half parts generate a new pic. This can increase training data.but I am not sure if it works. I think the iteration should be increased. because the this kind of augmentation indirectly increased the training data.

I've tried this kind of augmentation by erasing half of the aligned face to simulate face with mask. Loss decreased normally while accuracy was worse. My iteration was the same as baseline which may be the reson for this situation.

What's you test data? I think the test data should include such occluded face data which can verify the effectiveness of your data augmentations.

What's you test data? I think the test data should include such occluded face data which can verify the effectiveness of your data augmentations.
Just use LFW as fast validation. Acctually the experiment I did above may provide little information for u guys.
Recently I found my alignment method was too old (just adjust the eyes to the horizontal position and cut the face using the distance between two eyes) but not use Procrustes analysis as most articles did. This method cut too many pixels above eyebrows which may leave little information for masked face recognition.
Also, I've trained a network using cbamblock to do masked face recognition and the result on my maskface-dataset is ok (compare to my baseline without cbam). But the spatial part of cbam cannot be used in caffe (op tensor.max() cannot be realize in caffe prototxt), maybe I need to do some modification.

LFW should have a perfect score (99.83+%) very early on.. like after first LR drop.

Here I come again~
I use MaskTheFace to wear mask on my own face datasets. After finetune the dataset, I found the accuracy of maskface.VS.nomaskface got better while the normal situation accuracy dropped quickly. That means mask-face may have far different distribution compared with normal-face.

@cavalleria my results wouldn't be compatible as I use a different face detector and dataset(s).
I start with the fully trained baseline backbone and retrain with LR=0.01 + augmentations turned on
Some blur augmentations are good btw. I use a third-party package (kornia) to do Motion Blur (very effective!) and Gaussian Blur. I also have success improving accuracy with the augmentations in here.
I think the comparisons done in Model Zoo probably aren't very helpful though as augmented data is always going to perform worse than non-augmented with the same training time. It's more information for the model to learn.

Could you tell us what hyper-parameters you used in RandomMotionBlur if convenient? Thanks a lot!

I have tried to fine-tune the models with motion blur or gaussian blur, but I failed to get better accuracy. I will try them from scratch with p=0.1, any update will be posted here.

Unfortunately, it has not improved my performance:(

Very good discussion for data augmentation. I have some questions after reading the thread

Does motion blur and hflip improve performance? Other augmentation may decrease accuracy on normal face (LWF, Age30,...). Am I right?

For Masktheface augmentation, @luameows mentioned that performance on maskface increases but normal face is decreased. It means that model is overfitting to the maskface distribution. Do you agree

In conclusion, data augmentation may be not useful for face recognition task.

It depends on the distribution of the test data as to how well the augmentation will score.
Ultimately, if you have perfect frontal and high quality photos for train and test, you'll have over-fitted for this scenario. Adding augmentation like masks and blur will likely improve your model overall but do worse on your tests. There is only so much your model can encode until you go deeper.


Fix up tiny cases where user should not be nullable

These are cases where the logged in user could never be null and it's just a matter of telling the type system so.

There's some complexity to this because of an interaction with the server rendering that I don't 100% understand, so never mind  right now.


Cant connect

Issue Type: Bug
trying to connect but no way
Extension version: 1.0.2169
VS Code version: Code 1.45.1 (5763d909d5f12fe19f215cbfdd29a91c0fa9208a, 2020-05-14T08:33:47.663Z)
OS version: Darwin x64 17.7.0

System Info

Item
Value

CPUs
Intel(R) Core(TM) i5-2415M CPU @ 2.30GHz (4 x 2300)

GPU Status
2d_canvas: unavailable_softwareflash_3d: disabled_softwareflash_stage3d: disabled_softwareflash_stage3d_baseline: disabled_softwaregpu_compositing: disabled_softwaremetal: disabled_offmultiple_raster_threads: enabled_onoop_rasterization: disabled_offprotected_video_decode: disabled_offrasterization: disabled_softwareskia_renderer: disabled_off_okvideo_decode: disabled_softwareviz_display_compositor: enabled_onviz_hit_test_surface_layer: disabled_off_okwebgl: unavailable_softwarewebgl2: unavailable_software

Load (avg)
2, 2, 2

Memory (System)
8.00GB (0.38GB free)

Process Argv

Screen Reader
no

VM
0%

Thanks for reporting this issue! Can you please clarify the repro steps? What did you do, what you expected to happen, and what actually happen? Are you trying to start a new session or join another one? Are you behind a firewall?
Here is some info on connectivity in Live Share:
https://github.com/MicrosoftDocs/live-share/blob/master/docs/reference/connectivity.md

Closing due to age. Please feel free to reactivate with more details so we can repro the problem!


When downloading a Blob, failed requests (HttpResponseError exceptions) are not correctly retried

Package Name: azure-storage-blob
Package Version: 12.8.1
Operating System: Ubuntu 20.04
Python Version: 3.8

Describe the bug
We have an integration test that's using the following code
service_client = BlobServiceClient(
    account_url=f"https://{storage_account_name}.blob.core.windows.net",
    credential=os.environ["AZURE_STORAGE_ACCOUNT_KEY"],
)
service_client.get_blob_client(
    container=container, blob=blob
)
.download_blob()
.readall()

Sometimes, this test fails with this stacktrace (one line edited to remove a private filename, otherwise it's a copy-paste):
../.venv/lib/python3.8/site-packages/urllib3/response.py:438: in _error_catcher
    yield
../.venv/lib/python3.8/site-packages/urllib3/response.py:519: in read
    data = self._fp.read(amt) if not fp_closed else b""
/usr/local/lib/python3.8/http/client.py:458: in read
    n = self.readinto(b)
/usr/local/lib/python3.8/http/client.py:502: in readinto
    n = self.fp.readinto(b)
/usr/local/lib/python3.8/socket.py:669: in readinto
    return self._sock.recv_into(b)
/usr/local/lib/python3.8/ssl.py:1241: in recv_into
    return self.read(nbytes, buffer)
/usr/local/lib/python3.8/ssl.py:1099: in read
    return self._sslobj.read(len, buffer)
E   ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
../.venv/lib/python3.8/site-packages/requests/models.py:758: in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
../.venv/lib/python3.8/site-packages/urllib3/response.py:576: in stream
    data = self.read(amt=amt, decode_content=decode_content)
../.venv/lib/python3.8/site-packages/urllib3/response.py:541: in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
/usr/local/lib/python3.8/contextlib.py:131: in __exit__
    self.gen.throw(type, value, traceback)
../.venv/lib/python3.8/site-packages/urllib3/response.py:455: in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
E   urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
../.venv/lib/python3.8/site-packages/azure/core/pipeline/transport/_requests_basic.py:169: in __next__
    chunk = next(self.iter_content_func)
../.venv/lib/python3.8/site-packages/requests/models.py:761: in generate
    raise ChunkedEncodingError(e)
E   requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
tests/integration_tests/test_file:261: in _test_function
    service_client.get_blob_client(
../.venv/lib/python3.8/site-packages/azure/storage/blob/_download.py:520: in readall
    self.readinto(stream)
../.venv/lib/python3.8/site-packages/azure/storage/blob/_download.py:617: in readinto
    downloader.process_chunk(chunk)
../.venv/lib/python3.8/site-packages/azure/storage/blob/_download.py:129: in process_chunk
    chunk_data = self._download_chunk(chunk_start, chunk_end - 1)
../.venv/lib/python3.8/site-packages/azure/storage/blob/_download.py:211: in _download_chunk
    chunk_data = process_content(response, offset[0], offset[1], self.encryption_options)
../.venv/lib/python3.8/site-packages/azure/storage/blob/_download.py:52: in process_content
    content = b"".join(list(data))
../.venv/lib/python3.8/site-packages/azure/core/pipeline/transport/_requests_basic.py:188: in __next__
    raise HttpResponseError(err, error=err)
E   azure.core.exceptions.HttpResponseError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

When looking at the relevant code:
while retry_active:
    try:
        _, response = self.client.download(
            range=range_header,
            range_get_content_md5=range_validation,
            validate_content=self.validate_content,
            data_stream_total=self.total_size,
            download_stream_current=self.progress_total,
            **self.request_options
        )
    except HttpResponseError as error:
        process_storage_error(error)

    try:
        chunk_data = process_content(response, offset[0], offset[1], self.encryption_options)
        retry_active = False
    except (requests.exceptions.ChunkedEncodingError, requests.exceptions.ConnectionError) as error:
        retry_total -= 1
        if retry_total <= 0:
            raise ServiceResponseError(error, error=error)
        time.sleep(1)

the HttpResponseError is raised by the process_content line, but is not caught here. self.client.download doesn't raise an exception, it returns an iterator instead. The exception is raised when process_content iterates on response.
To Reproduce
Steps to reproduce the behavior:

Run many times a code snippet similar to the above, until a "connection reset by peer" network error occurs (I haven't wrote a simulator for that, but this occurred a few times in out integration tests so far).

Expected behavior
The retry code is triggered, and the network error does not surface to the application.
Screenshots
N/A
Additional context
Maybe I'm misinterpreting the stack trace here. The code never configures retry parameters, it uses the default provided by the Azure SDK. To avoid encountering this issue, should I change the retry parameters instead?

@tasherif-msft can you take a look?

@jalauzon-msft fyi

Hi @Neki we are aware of this issue. These resets are triggered by the service. We are currently in discussions on handling these types of errors the Python SDKs. We will update you once we have more information

Hi @Neki, I've marked this Issue for close as I think retries are properly occurring in the latest versions, as mentioned. We can use #19753 to track any Connection Reset errors that are getting through the retries. There is also an example of how to increase retry count there.


adds Group resource

Adds resource management for Group resources in the IAM API.
Signed-off-by: Jay Pipes<EMAIL_ADDRESS>By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

@A-Hilaly @RedbackThomson OK, updated. thanks for the reviews!

/retest

/retest


small z3 improvement

This small change just checks if a given composite n is perfect square or not, if is not it proceeds to SMT the factorization given that p>isqrt(n)  and q<isqrt(n) where n=pq.
Test ran ok
./RsaCtfTool.py --publickey z3.pub --attack z3_solver --private
[] Testing key z3.pub.
[] Performing z3_solver attack on z3.pub.
Results for z3.pub:
Private key :
-----BEGIN RSA PRIVATE KEY-----
MBsCAQACAQ8CAQMCAQMCAQMCAQUCAQECAQMCAQI=
-----END RSA PRIVATE KEY-----

Cool !
You can add your test in the new test() method in z3 attack file, test.sh is removed and it will be the only testing method implemented
./RsaCtfTool.py --attack z3_solver --tests

Will do
Cheers


DataFrame.fromCsv FileNotFoundException When Asset Is actually found?

I can create a tmp file to prove that the asset is working, but the .fromCsv function appears to break. What am I doing wrong here?
Future<void> loadUSZIPCodesCSV() async {
String path = 'assets/files/uszips.csv';
final file = File(path);
var existsSync = file.existsSync();
print('File path is actually: ${file.path} and ${existsSync}');
final df = await DataFrame.fromCsv(file.path,
dateFormat: "MMM dd yyyy", verbose: true);
df.show();
print(df.columns);
}

My log is:
I/flutter (24093): File path is actually: assets/files/uszips.csv and false
E/flutter (24093): [ERROR:flutter/lib/ui/ui_dart_state.cc(157)] Unhandled Exception: Instance of 'FileNotFoundException'
E/flutter (24093): #0      DataFrame.fromCsv (package:df/src/df.dart:124:7)

print('File path is actually: ${file.path} and ${existsSync}');
I/flutter (24093): File path is actually: assets/files/uszips.csv and false

your file does not exist

But it does... I'm not sure what is wrong

The issue was resolved by leveraging rootBundle:
 //Read csv
   String path = 'assets/files/uszips.csv';
   ByteData data = await rootBundle.load(path);
   String directory = (await getTemporaryDirectory()).path;
   File file = await writeToFile(data, '$directory/uszips.csv');


New action request:

Action name
devops-actions/issue-comment-tag
Reasons
Test Reason

Intended usage
Test

I have checked the internal marketplace for similar actions and couldn't find one that works for us.

[X] I have checked the internal marketplace first

The tagging works! 🤣


add multi-cursor

You missed the Table Of Contents.

Closing because never adjusted and now it requires conflict resolution. Please re-open if addressed.


Wiki - Long Running Operations

The Long Running Operations section of the Java repo wiki is currently empty.
https://github.com/Azure/azure-sdk-for-java/wiki/Long-running-operations
LRO conceptual topic should be completed similar to the topic of pagination https://github.com/Azure/azure-sdk-for-java/wiki/Pagination-and-Iteration

We now have https://docs.microsoft.com/en-nz/azure/developer/java/sdk/lro


x error 8 (BadMatch) on Linux Mint

Opening an 800x600 window from piston_window with GL version 2.1, and rendering the text "Hello, World!" gives the following X-error, though everything seems to have worked fine:
[glutin] x error code=8 major=42 minor=0!
Based on this page it appears that a bad argument is being passed to an xlib function somewhere. (Error code 8 -> BadMatch, described as a "parameter mismatch".)
I can't investigate till later, but I figured I'd open this issue now.

I see the same message on stdout every time I run Servo (on Archlinux x86_64 with GNOME 3), but rendering still seems ok.
There’s probably two parts to this issue:

Fixing the underlying cause, which might be a "parameter mismatch"
Should these messages really go to stdout by default? I see in src/api/x11/xdisplay.rs:

        unsafe extern "C" fn x_error_callback(_: *mut ffi::Display, event: *mut ffi::XErrorEvent)
                                              -> libc::c_int
        {
            println!("[glutin] x error code={} major={} minor={}!", (*event).error_code,
                     (*event).request_code, (*event).minor_code);
            0
        }

        unsafe { (xlib.XInitThreads)() };
        unsafe { (xlib.XSetErrorHandler)(Some(x_error_callback)) };

But I don’t think numeric error codes from X11 are very meaningful to developers using gultin, much less end users. What do you think of having x_error_callback be a no-op by default, unless logging is enabled somehow? Maybe with https://crates.io/crates/env_logger/ ?

@SimonSapin The meaningfullness of X11 error messages has been slightly improved by #675 (it now retrieves the error description from xlib).
Other than that, I agree with you regarding error reporting. Maybe, at least, the errors should be written to stderr instead of stdout ?

I added better error handling for X in #679
As for the actual error, it needs more diagnostic (with #679). Most of the errors I see come from this code: https://github.com/tomaka/glutin/blob/8236564a5224e38cd6e345df8ffaf343bfa24320/src/api/glx/mod.rs#L179-L199
It tries to create a context with version 3.2. If version 3.2 is not supported, an error is returned and it tries to create a context with version 3.1. If an error is returned again, it tries version 1.0. This is a normal process, but every time there is an error it gets printed to stdout.

#679 looks good to me.

The error message I get with #679 is "Failed to call XSendEvent after wakeup", which is api::x11::WindowProxy::wakeup_event_loop, not api::glx::ContextPrototype::finish.

This doesn't look relevant anymore, and even if it were, it would now be a problem with winit.


Remove macro def

It has been finally removed from the compiler in https://github.com/crystal-lang/crystal/pull/5040.

Thx!


Private messages upon install

I got 3 messages from people advertising their discord after I installed this app

This app does not cause that, if you joined the BD servers, it could be from spambots there. You should report them to discord and to the admins of the BD servers so they can be banned.


clear_session() with tensorflow 1.8 backend results in a segfault

Hi,
the following code results in a segfault after the model was fitted a second time:
import numpy as np
from keras.models import Sequential
from keras.layers import Dense
from keras.backend import clear_session

def test():
    data = np.random.random((1000, 100))
    labels = np.random.randint(2, size=(1000, 1))
    model = Sequential()
    model.add(Dense(32, activation='relu', input_dim=100))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(optimizer='rmsprop',loss='binary_crossentropy',metrics=['accuracy'])
    model.fit(data, labels, epochs=3, batch_size=32)

test()
clear_session()
test()

The output:
Using TensorFlow backend.
Epoch 1/3
1000/1000 [==============================] - 3s 3ms/step - loss: 0.7174 - acc: 0.4910
Epoch 2/3
1000/1000 [==============================] - 0s 72us/step - loss: 0.7056 - acc: 0.5050
Epoch 3/3
1000/1000 [==============================] - 0s 70us/step - loss: 0.6989 - acc: 0.5150
Epoch 1/3
1000/1000 [==============================] - 0s 187us/step - loss: 0.6936 - acc: 0.5320
Epoch 2/3
1000/1000 [==============================] - 0s 67us/step - loss: 0.6854 - acc: 0.5600
Epoch 3/3
1000/1000 [==============================] - 0s 70us/step - loss: 0.6802 - acc: 0.5720
Segmentation fault (core dumped)

Not calling clear_session() prevents the segfault. This only happens with keras 2.2 and tensorflow 1.8 (both CPU and GPU version). All other combinations of keras (2.1.x) and tensorflow (<1.8 and also 1.9.0rc0) don't result in a segfault.
Is this a keras or tensorflow issue?

I had the same issue as @budach . I had to downgrade to tensorflow-gpu 1.8 with cuda 9.0 and cuDNN 7, in order to avoid this error with Keras 2.2.0.

tl;dr:

No issues when using keras 2.1.6
No issues when using tensorflow-cuda
No issues when using tensorflow and python2.7
Segfault appears after https://github.com/keras-team/keras/pull/10087
The only combination that gives segfaults for me is python3.6, tensorflow, keras 2.2.0

My unit tests on Travis were failing because of this .. I'll explain my situation, perhaps it adds some information. My python2.7 build was finishing successfully, python3.6 was failing with a segfault. Locally, using the GPU version of tensorflow, everything passes on python3.6 (regardless of actually using a GPU or not). The code I was using is similar to that of the original author of this issue:
import keras
import numpy as np
import tensorflow as tf

def get_session():
    config = tf.ConfigProto()
    config.gpu_options.allow_growth = True
    return tf.Session(config=config)

def do_train():
    data = np.random.random((1000, 100))
    labels = np.random.randint(2, size=(1000, 1))
    model = keras.models.Sequential()
    model.add(keras.layers.Dense(32, activation='relu', input_dim=100))
    model.add(keras.layers.Dense(1, activation='sigmoid'))
    model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])
    model.fit(data, labels, epochs=1, batch_size=1)

if __name__=='__main__':
    keras.backend.tensorflow_backend.set_session(get_session())
    do_train()

    keras.backend.tensorflow_backend.set_session(get_session())
    do_train()

But I think the underlying issue is the same. I've also run it with gdb, which gives me the following:
#0  0x00007fffd6ba3398 in tensorflow::TF_SessionReleaseCallable(TF_Session*, long, TF_Status*) () from /home/hgaiser/dev/keras-segfault/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#1  0x00007fffd6b525e2 in _wrap_TF_SessionReleaseCallable () from /home/hgaiser/dev/keras-segfault/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#2  0x00007ffff7420343 in _PyCFunction_FastCallDict () from /usr/lib/libpython3.6m.so.1.0
#3  0x00007ffff73e984e in ?? () from /usr/lib/libpython3.6m.so.1.0
#4  0x00007ffff73ad0fa in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.6m.so.1.0
#5  0x00007ffff73e8b2b in _PyFunction_FastCallDict () from /usr/lib/libpython3.6m.so.1.0
#6  0x00007ffff741b28f in _PyObject_FastCallDict () from /usr/lib/libpython3.6m.so.1.0
#7  0x00007ffff741bc12 in _PyObject_Call_Prepend () from /usr/lib/libpython3.6m.so.1.0
#8  0x00007ffff741afec in _PyObject_FastCallDict () from /usr/lib/libpython3.6m.so.1.0
#9  0x00007ffff73ef136 in ?? () from /usr/lib/libpython3.6m.so.1.0
#10 0x00007ffff7381b61 in ?? () from /usr/lib/libpython3.6m.so.1.0
#11 0x00007ffff7480b9d in ?? () from /usr/lib/libpython3.6m.so.1.0
#12 0x00007ffff7480c02 in PyGC_Collect () from /usr/lib/libpython3.6m.so.1.0
#13 0x00007ffff748496f in Py_FinalizeEx () from /usr/lib/libpython3.6m.so.1.0
#14 0x00007ffff7481313 in Py_Main () from /usr/lib/libpython3.6m.so.1.0
#15 0x0000555555554b5c in main ()

I've binary searched which commit was to blame and found https://github.com/keras-team/keras/pull/10087 to be the culprit.
@fchollet @TimZaman @ahundt can you shed some light on this issue?

Hey Hans. Any reason you set the session multiple times with your custom function? Could you tru calling the backend's clear_session() in between setting the sessions?

Hey Tim :)

Any reason you set the session multiple times with your custom function?

I don't particularly want to do this, but we are running unit tests on our training script. There are some parameters that we want to test (different models), so the training script is tested multiple times. Every time the training script is executed it sets a tf session. I could write it such that it only does that if you execute it directly (by moving it to if __name__=='__main__':), but I suppose there shouldn't be anything wrong with setting a session multiple times.

Could you tru calling the backend's clear_session() in between setting the sessions?

I could, but this will still segfault. As @budach pointed out, the issue also occurs when doing the sequence "training -> clear_session -> training". I noticed the same behaviour if I replace set_session with clear_session in my examples.

Ok. Did you try with the unit test decorator for keras?
On Wed, Jun 27, 2018, 09:47 Hans Gaiser<EMAIL_ADDRESS>wrote:

Hey Tim :)
Any reason you set the session multiple times with your custom function?
I don't particularly want to do this, but we are running unit tests on our
training script. There are some parameters that we want to test (different
models), so the training script is tested multiple times. Every time the
training script is executed it sets a tf session. I could write it such
that it only does that if you execute it directly (by moving it to if
name=='main':), but I suppose there shouldn't be anything wrong
with setting a session multiple times.
Could you tru calling the backend's clear_session() in between setting the
sessions?
I could, but this will still segfault. As @budach
https://github.com/budach pointed out, the issue also occurs when doing
the sequence "training -> clear_session -> training". I noticed the same
behaviour if I replace set_session with clear_session in my examples.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/keras-team/keras/issues/10399#issuecomment-400749880,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHXSRJRllQn7Mdm4MEJesLytJpNAmxlSks5uA7cpgaJpZM4Uibsw
.

I have the same issue with tensorflow 1.8.0 and keras 2.2.0

For what it's worth, I have the same problem with  tensorflow 1.8.0 and keras 2.2.0. I had placed clear_session at the end of each fold in a cross-validation loop and was gettingSegmentation fault (core dumped). Removing clear_session resolved this issue.

upgrading tensorflow-gpu with pip resolved the problem for me.

thanks @hgaiser , I had Keras-2.2.2 installed and downgrading to 2.1.6 solve the problem of Setmentation fault when clear_session

Closing as this is resolved


Update staticfile.rs Example

I'm interested in using tide, but I kinda really need to be able to serve static files. The archived example was great, but it's now outdated and doesn't compile. Could we work on updating this?
Thanks!

I did a rough translation of the old example here, though I'm not sure it's in good enough shape to paste directly in.

@eignnx I'm thinking we should probably expose APIs for static file serving directly; I was never too happy with the old example, and it felt like something we should be providing a workflow for.
Ideas / constraints on what we'd like to handle would be very welcome!

In case anyone else comes across this issue thread, I've released my example implementation as a crate here: tide-naive-static-files. While this issue is being worked on, you may find it useful.

I was the author of the old staticfile example. I finally got some time to update staticfile code in my server, which maybe useful for @eignnx or anyone else.


[#1398] fix(MR)(TEZ): Limit attemptId to 4 bit and move it from 18 bit atomicInt to 21 bit taskAttemptId in 63 bit BlockId.

What changes were proposed in this pull request?
Limit attemptId to 4 bit and move it from 18 bit atomicInt to 21 bit taskAttemptId in 63 bit BlockId.
Fix blockId overflow problem.
Why are the changes needed?
Fix: #1398
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Existing UT and integrated tests.
This is a draft PR, I'm still doing the rest of the testing.

Codecov Report
Attention: 9 lines in your changes are missing coverage. Please review.

Comparison is base (602611c) 53.23% compared to head (239c418) 54.12%.

Files
Patch %
Lines

...n/java/org/apache/hadoop/mapreduce/RssMRUtils.java
50.00%
1 Missing and 4 partials :warning:

...c/main/java/org/apache/tez/common/RssTezUtils.java
55.55%
0 Missing and 4 partials :warning:

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #1418      +/-   ##
============================================
+ Coverage     53.23%   54.12%   +0.88%
  Complexity     2721     2721
============================================
  Files           419      399      -20
  Lines         23982    21607    -2375
  Branches       2045     2041       -4
============================================
- Hits          12768    11694    -1074
+ Misses        10427     9197    -1230
+ Partials        787      716      -71

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@jerqi Could you please provide suggestions on areas that need improvement?

cc @zhengchenyu could you help review this ?

Sine https://github.com/apache/incubator-uniffle/pull/1529 is merged into master, I think we should review this PR?
After this PR, the blockid calculation for spark, mr, tez will remain consistent. Then we will reduce the probability of overflow problems.
@qijiale76 Can you reconstruct the code according to https://github.com/apache/incubator-uniffle/pull/1529?

@qijiale76 Do you want to push this forward?

@qijiale76 Do you want to push this forward?

Yes, I’ll reconstruct the code next week.

@EnricoMi Thanks for your very helpful review. I have updated this PR based on your suggestions and by referring to Spark's implementation. Could you please review the latest code again?

@EnricoMi, thank you very much for your reviews. I have reverted taskAttemptIds to long in the MR, Tez, and Spark engine clients. Could you please review this PR again?

@jerqi I still think taskattemptid should be int. Since we've limited taskattemptid to no more than 32 bits, int is enough. For spark, taskattemptid has changed, now does not comes from TaskContext::taskAttemptId.
See the code:
https://github.com/apache/incubator-uniffle/blob/f7c6d2da237bd487d3cd0e21231108df90559cbe/common/src/main/java/org/apache/uniffle/common/util/BlockIdLayout.java#L69

Code paths that work fine with long taskAttemptIds should show that via using long, paths that are limited to intids should indicate that usingint`. Then you know what the code supports and which code has limitations.

LGTM+1


Update browserslist dependency to fix leaked global

In the 0.2.0 version of browserslist there's a leaked global, normalizeVersion. It was fixed in 0.3.1but dependency in autoprefixer-core is locked to ~0.2.0.

I can update it only in minor release, not in patch releases. Because Browserslist 0.3 has new features.
I will be able to write Autoprefixer 5.2 only afte I will finish PostCSS Plugin Guidelines.


Conflict detection/avoidance for "require" aliases

React 16 bundles now use rollup as well; this means some of them may contain aliased requires, which will clash with the ones generated by the local rollup.
Example:
React <IP_ADDRESS>, node_modules/react/cjs/react.development.js
var objectAssign$1 = require('object-assign');
var require$$0 = require('fbjs/lib/warning');

gets converted to
var objectAssign$1 = require$$0;
var require$$0 = warning_1;

, which is obviously invalid.
This situation could be avoided if the name candidates were tested for uniqueness while increasing the uid.
Alternatively, either the separator ($$) or the start uid (0) would be configurable in transform.js .

I have attached a small project that exhibits this issue. The production build does not go through the code block that contains the conflicting variable.

$ yarn --version
1.1.0
$ nodejs --version
v6.11.3
$ npm --version
3.10.10
$ yarn install
(...)
$ npm run-script build

<EMAIL_ADDRESS>build
rollup -c
./src/index.tsx → ./dist/bundle.js...
created ./dist/bundle.js in 3.3s

repro.zip

Tiny test case here: https://github.com/rollup/rollup/issues/1655#issuecomment-332337049

Hi @mbostock et al, as it appears that @Rich-Harris is currently on an extended vacation with limited internet access and cannot release a new version to npm, I have set up a make-shift solution for anyone suffering from this bug. You can now
npm install<EMAIL_ADDRESS>
which should contain all necessary build artefacts.

Any ETA on an npm release? It took a lot of googling to get here.

Unfortunately not. It seems @Rich-Harris is on extended vacation with limited internet access and I do not know who else can publish to npm. Until then I fear you need to stick with the github install.

Work perfectly for me ! Thank for the set-up.
I hope this is not vacation around the world ?!

It seems like 8.2.4 was published yesterday on NPM so it is now fine :)

I'm getting
TypeError: Object(...) is not a function
now, is this related?


[BUG] resource android:attr/dialogCornerRadius not found.

After I added your library, I got this bug.
error: resource android:attr/dialogCornerRadius not found.
Message{kind=ERROR, text=error: resource android:attr/dialogCornerRadius not found., sources=[C:\Users\luiss\.gradle\caches\transforms-1\files-1.1\appcompat-1.0.0-beta01.aar\c50acad9c7755b75bb700a742378cb2b\res\values-v28\values-v28.xml:9:5-12:13], original message=, tool name=Optional.of(AAPT)}

I set  this to my app gradle
  implementation 'com.divyanshu.colorseekbar:colorseekbar:1.0'
Did I set anything wrong ? And how can I fix it ?
Thank you in advance  !

@thanhthein The library is using Androidx. Try using android studio 3.2+ and migrate your app to Androidx. Let me know if it works. I'll be making changes to fix the bug soon, till try above thing.


Update error message of Update-Help when culture is C.UTF8

Prerequisites

[X] Write a descriptive title.
[X] Make sure you are able to repro it on the latest released version
[X] Search the existing issues.
[X] Refer to the FAQ.
[X] Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

Install Ubuntu20 with default culture (C.UTF8)
Update-Help

For more details: https://github.com/PowerShell/PowerShell/pull/18037#discussion_r1187698068
Related issue: https://github.com/PowerShell/PowerShell/issues/6217
Expected behavior
Error message saying "Your culture is C.UTF-8, which is a default not associated with any language, consider changing your system culture or install the English-US help content using: Update-Help -UICulture en-US."

Actual behavior
Update-Help: Failed to update Help for the module(s) 'Microsoft.PowerShell.Core' with UI culture(s) {C}:
Unable to retrieve the HelpInfo XML file for UI culture C.
Make sure the HelpInfoUri property in the module manifest is valid or check your network connection and then try the command again..
English-US help content is available and can be installed using: Update-Help -UICulture en-US.

### Error details

```console
NA

Environment data
Name                           Value
----                           -----
PSVersion                      7.3.4
PSEdition                      Core
GitCommitId                    7.3.4
OS                             Microsoft Windows 10.0.22621
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

Visuals
NA

Updated expected error message as per WG discussion.


Tray icon blank under Windows 11

As is [in]visible on this screenshot (circled in red).

Hey @cbouc just published a new release with the 2 fixes
Thanks for reporting theses problems!

Thank you for the (fast!) fixes, I now see the icon in the tray and the dashboard is restored by the menu when it’s minimized!
From: Colin Lienard @.>
Sent: lundi 20 mai 2024 19:14
To: colinlienard/gitlight @.>
Cc: Christophe BOUCHON @.>; Mention @.>
Subject: Re: [colinlienard/gitlight] Tray icon blank under Windows 11 (Issue #283)
Hey @cbouchttps://github.com/cbouc just published a new release with the 2 fixes
Thanks for reporting theses problems!
—
Reply to this email directly, view it on GitHubhttps://github.com/colinlienard/gitlight/issues/283#issuecomment-2120861897, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABKVPFXEPQJFXFJN7AFFMLLZDIVO5AVCNFSM6AAAAABH2TU7CWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRQHA3DCOBZG4.
You are receiving this because you were mentioned.Message ID: @.***>


experiment with QuickJS environment variables

With Wasi and QuickJS we might be able to use a more mature/less hacky environment variable implementation

Got it working!


WFLY-17561 Move test dependency which is dependent on the WFCORE

Issue: https://issues.redhat.com/browse/WFLY-17561
Related WildFly Core issue: https://issues.redhat.com/browse/WFCORE-6203

Hello @yersan,
Previously, I was not aware that we can use test bom in every module that's why some test maven dependencies were left. I am working on it, I will update the code soon.
Thanks

The one CI job fails due to https://issues.redhat.com/browse/WFLY-17702. I'll fix that.

/retest

Sure @bstansberry.

Thanks @parsharma and @yersan !!!!!


devcontainer-cli: init at 0.53.0, nodePackages.node-pty: init at 1.0.0

Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/2994

Please use buildNpmPackage for the new node package.

Would you mind elaborating on this? Are packages no longer supposed to go into pkgs/development/node-packages?

Are these kinds of development node packages no longer supposed to go into pkgs/development/node-packages?

Yes, because it is a giant mess and very hard to maintain and generate.

@mr360

Would you like to combine forces as we are essentially tackling the same problem :-)

Sure :)


Upgrade to react-router v4

So much more beautiful and solves issues with RHL.

I am going to wait for the SSR story for react-router 4 to flesh out a bit...

Mark Dalgleish is doing a fantastic job pushing this story on the related react router repo.  If you are interested track the issues there:
https://github.com/ReactTraining/react-router-addons-routes

Boom! There has been some movement on the dependencies I was waiting for.  I am going to do try the shift soonish.

I have a working proof of concept for this on the next branch! :)
It still needs some tidying up, restructuring and documentation updates, but it is looking promising. :)

Had a look at some of the work on the next branch. Awesome stuff!
I found a couple of examples where some people are using v4 with code splitting.
https://github.com/gabrielbull/react-router-server
...and a much larger production example that also uses hot module reloading
https://github.com/LWJGL/lwjgl3-www
Not saying that your method is wrong at all, but I just stumbled across these and thought they may be useful as you develop this branch. As a frame of reference. If not, feel free to ignore them.

That's awesome thanks!  I'll definitely look into it.  I am not 100% set yet.  Still lots of work to do. :)

@carsonperrotti Please do let me know if you have any personal preference, and why.  It's great to hear other peoples opinions. :)

@carsonperrotti I've done another round of updates to this next branch.
The entire component structure has been restructured to better represent the local route declaration that react router 4 provides us.
I've also solved the issue with displayed a consistent 404 page when hitting a missing route with or without server side rendering enabled.
Thanks for the links earlier, but I have decided to stick with my method of supporting code splitting with react router 4.  I feel like it's a bit simpler than the solutions within those links.  I would like us to always use the simplest solution available for each feature, allowing users to upscale into more complex solutions as and when they feel their projects have matured to the point that it is needed.
I would appreciate it if you could have a look see and let me know if your like or dislike anything.
The next thing on my plate is to provide the most simple example of making sure your app is hydrated with required data before rendering - especially useful for SSR projects.

Thanks for the info on the update. I too agree that your solution is the most simple. I spent the weekend hacking away at a few options in my local fork but just reverted to the last commit after a little.
One thing I have done differently, I have integrated (Apollo)[http://apollostack.com] into my fork and use that for data fetching and rehydration, but it's not too different from any Redux SSR implementation.
I'll check out the update now. Thanks!

I really need to check out Apollo Stack, not the first time its been mentioned to me.

I think I just about have a decent data prefetch example that is fairly simple.  It would be interesting to hear if the same structure would satisfy the requirements of other data libraries/frameworks such as the apollo stack.

Loads more improvements and enhancements to the next branch.  Think I have just about fleshed out data prefetch requirements.

Moved all the data fetching logic to the redux branch.


Automated cherry pick of #2744: qcloud sunday

Cherry pick of #2744 on release/2.11.
#2744: qcloud sunday

Codecov Report

:exclamation: No coverage uploaded for pull request base (release/2.11@eae568c). Click here to learn what that means.
The diff coverage is 0%.

@@              Coverage Diff               @@
##             release/2.11   #2745   +/-   ##
==============================================
  Coverage                ?   7.22%
==============================================
  Files                   ?     470
  Lines                   ?   74384
  Branches                ?       0
==============================================
  Hits                    ?    5376
  Misses                  ?   68557
  Partials                ?     451

Flag
Coverage Δ

#aFlag
7.22% <0%> (?)

Impacted Files
Coverage Δ

pkg/compute/models/snapshotpolicycache.go
4.32% <0%> (ø)

pkg/compute/models/snapshotpolicydisks.go
0.39% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eae568c...b24ab56. Read the comment docs.


Add regex filtering

Instead of matching on a fixed string we should be able to match on a regex

Agreed! Had considered using the regex crate for this but haven't gotten around to it yet. Will do, this weekend at the earliest.
Thanks!

Added in 3d9427b. Tested using a "pop pop ret" regex: ^(?:pop)(?:.*(?:pop))*.*ret.


Can someone help me understand this issue

The code below I used to build the dataframe you see in the image for some FX data. Problem is the last values for the SMA data is not in same row as the latest price data. Daily data should be based on the same time, especially FX since markets closes at 5pm EST and there is no activity Sat & Sun.
Any idea how to fix this?
 data_fx, meta_data_fx = fx.get_currency_exchange_daily(from_symbol=fx_from,to_symbol=fx_to,outputsize='compact')    data_sma30, meta_data30 = ti.get_sma(symbol=fx_from + "" + fx_to,interval="daily",time_period=30,series_type="close")    data_sma50, meta_data50 = ti.get_sma(symbol=fx_from + "" + fx_to, interval="daily", time_period=50, series_type="close")    fx_df = data_fx    sma30_df = data_sma30    sma50_df = data_sma50
'total_df = pd.concat([fx_df,sma30_df,sma50_df], axis=1, sort=True)`

Temporarily closing this issue due to a prolonged period since the last response was posted. Will re-open if needed.


How can I convert a signed raw Fxp representation back to a float?

I need to convert Fxp processed raw, signed float representations back to floats in (say) a CUDA kernel (I can't use Fxp for this reverse conversion). For unsigned representations,
floatVal = rawVal / (2 ^ 8)
works fine and matches up with the expected value. However, I see no information on how Fxp deals with signed values. How can I convert a raw representation of a signed float back into a signed float?
Thanks!

I figured it out - leaving a link to the solution here for anyone else that might want to do the same. Since I'm using cupy, I wrote this for numpy so the answer is broadly applicable.
Gist with code: https://gist.github.com/aniongithub/50050fc192ac1a16e144bc6de51252db

Hello @aniongithub
It's great that you found the way to figure it out! You're right about calculate the two's complement before conversion.
Let me left some code by here to resolve that procedure using fxpmath:
from fxpmath import Fxp
from fxpmath.utils import twos_complement_repr

float_vals = [1.25, -2.5]
x = Fxp(float_vals, signed=True, n_word=32, n_frac=8)
uraw_vals = x.uraw() # get the unsigned raw value, you could cast to uint32 if you want doing x.uraw().astype(np.uint32)

# to recover the float value you could do:
r = Fxp(twos_complement_repr(uraw_vals, x.n_word), raw=True, like=x)
# you can change `x.n_word` to amount of bits of raw binary representation

# checking both are the same
assert (x == r).all()


dotNet not opening

Can't open my dotNet on my desktop
Everything looks fine and insalled yet no response

There is not enough information to act on this issue. Please provide more info (including repro steps and error messages)

Have you opened a new terminal/command line after installation?


Add ability to customize node display

I'm trying to figure out a way to highlight a node (add a class) when the user clicks on it.  I have an onClick handler, but I only have access to the node, not to the display element, nor do I have a way to customize the template of a node.  Any suggestions on how I can do this?  If it matters, I do not need checkboxes, so have them disabled, so could use the isSelected state for marking a node as needing a highlight.

Closing this as a duplicate of #25 which, good news, is already on our short list of items for v1.0.0.
There's currently no supported way to accomplish what you're after (hopefully soon!) but in the near term you can work around this with a (reasonably horrible) hack by overloading the ivh-treeview-node directive or tapping the ivh-treeview-node-label class with a directive of your own making: example.
Side note: I also added to our v1.0.0 list to document which classes shold be expected on what elements.

Thanks, I'll give that a shot


improve repertoire selection

Original design was for small # of repertoires, that seems to be common but not always correct; e.g., with pharma projects that can have 80+ samples. Try for a model that stays simple like now for small list but expands better.

cap the vertical use of space with scrollbar
Implement some kind of shift-click or select all

added vertical scroll if list gets > 300px and implemented ctrl-a to select all in list.


read: fix panic on witness check

Signed-off-by: Connor1996<EMAIL_ADDRESS>
What is changed and how it works?

Issue Number: Close #13764
What's Changed:

After the remove peer conf-change is applied and before the peer is destroyed.
There is a chance that local reader may not find the peer from region info.
So this PR considers this case and fixes panic on witness check.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List
Tests

No code

Release note
None

/merge


Is there an error in the self-attention expression?Eq(1)

When calculating the time complexity of Self-attention, is the expression of Q = WQ · X reversed? At the same time, the dimension of X should also be reversed?

The ’error‘ exits in Eq.(1).

It's a typo. We already corrected in the camera ready.


experiment(web): Remove component spinner on update

We are experimenting with the removal of the spinner icon for component upgrades and changing the colour and opacity of the component

bors merge


Detaching STN input

Hello, can I ask you a question?
Why the input of conv1 in SpatialTransformer should be detached?
Look forward to your reply.
Originally posted by @wuwenbin970731 in https://github.com/aicaffeinelife/Pytorch-STN/issues/5#issuecomment-659167815

That's a good question. I don't know why it's there but I implemented STN myself based on this repository without detaching the input and it seems to work fine.

Thanks for your reply, I  will try it without detaching the input.


ticker: Add issuer addresses to markets.json

https://ticker.stellar.org/markets.json returns pairs with this shape:
        {
            "name": "ABDT_USD",
            "base_volume": 0,
            "counter_volume": 0,
            "trade_count": 0,
            "open": 10,
            "low": 10,
            "high": 10,
            "change": 0,
            "base_volume_7d": 0.0001401,
            "counter_volume_7d": 0.0000141,
            "trade_count_7d": 1,
            "open_7d": 10,
            "low_7d": 10,
            "high_7d": 10,
            "change_7d": 0,
            "price": 10,
            "close": 10,
            "close_time": "2019-06-14T16:48:12Z",
            "bid_count": 1,
            "bid_volume": 0.1394209,
            "bid_max": 0.0001,
            "ask_count": 4,
            "ask_volume": 9286.454205,
            "ask_min": 0.1,
            "spread": 0.999,
            "spread_mid_point": 0.4996
        },

But it's not possible to tell if this is AnchorUSD's token, or token.io's, or some random bozo's.

@morleyzhi markets.json aggregates trade data by asset codes, regardless of issuer. You can use the markets() query on the ticker's GraphQL interface (https://ticker.stellar.org/graphiql) to get trade data by specific trade pairs.


Refresh and Simplification of QSG README

Update of QSG that focuses primarily on:

Removal of redundant install information provided in the main SRF README
Improved organization (added TOC and streamlined prose)
Typo fixes

Targeting branch 22.08 with this, but we should also merge to 22.06

@gpucibot merge


Emojis still don't load in production /generate

I'm using my custom buildpack (https://github.com/gannicottb/puppeteer-heroku-buildpack.git) which is based on (jontewks/puppeteer)
heroku buildpacks:remove jontewks/puppeteer
heroku buildpacks:add https://github.com/gannicottb/puppeteer-heroku-buildpack.git
Anyway the point of that was to add fonts-noto-color-emoji so the emojis would load through puppeteer.
But we're still getting rectangles, sadly.
I'm not sure whether digging deeper into this is worth it since emojis are just supposed to be a hack until we have custom icons loaded through some other means.

We're using React Icons for Potion Sellers now, so this is a non issue.


Add Options to all Operations

The Operation interface should be updated to implement Options to add options to all Operations.
This is required for the FederatedStore #955 .

Migration Steps
All operations must now have an options field. The Operation interface has been updated to include a getter and setter for options. The Options interface has been removed.
When added the options field remember to also add it in the shallowClone method.

Merged into develop.


Existant ava browser runner

Is your work based on this ?
https://github.com/avajs/ava/pull/3086/files

No, but it is only a simple proof of concept with minimal functionality.

are you planning to do more ?
I have some issues & features request with my first try.

If you plan to push it in npm for more use than proof of concept, i would be happy to contribute.

Sounds great, I have no problem publishing the module to npm

Who ! You work on it so hard ! It's marvelous !

we can, but we also can ask https://github.com/avajs to add it for extra visibility


Hello a problem in Django on inux

Issue Summary
Hello a problem in Django on inux
Hello
When I'm trying to execute this command while installing Graphite
/usr/lib/python3/dist-packages/django/bin/django-admin.py migrate --settings=graphite.settings
I get this message !!
No migrations to apply.
Your models have changes that are not yet reflected in a migration, and so won't be applied.
Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
Please I really need your help

Hi there. We use Github only for managing bugs and feature requests, and not for support. We recommend that you ask for help on Stack Overflow using the django-oscar tag, or joining our slack workspace or mailing list and asking a question there. Thanks.


Search cmd

Notes for Reviewers
This PR fixes #9844

[x] Implement search command with filters model name, kind, subtype, type
[ ] Develop test cases to ensure command functionality

Signed commits

[x] Yes, I signed my commits.

A low priority.
When will the model import logs land? It's been many months now.

A low priority.
When will the model import logs land? It's been many months now.

I'll get to it I saw ripul had an open pr already had a look to it saw your comments too will start working on it . Is there anything before hand that I must be aware of @leecalcote
You are talking about this pr

Ready?

Ready?

No need to add the check for case insensitive flags.


fix ingress root path and order

Some ingress controllers don't treat paths without * as prefix types in v1beta. Also moved the /* to the end as the catchall

Hi @tybritten which ingress controllers has this been tested against?

I had tested it against alb and traefik but now it doesn't work on nginx if you're on 1.19+...might be time to just switch to v1. The current config doesn't work with alb

Switching to v1 makes sense to me, ingress has been out of beta for a while

ok updated


Chunk Loading | ReOpen.

Please view this thread or close this an reopen this one: https://github.com/MylesIsCool/ViaVersion/issues/115

We are aware of the original issue, do not open more issues.


Update current script color on change

Update current script color on change as mentioned in #5450

Not work when "Script temperature enabled" option is disable.


Override location of opensc when using nix

Description
Nix installs opensc which we depend on in prime-api-client. However, the default location that prime-api-client looks for the library where homebrew installs it. This PR sets an override when nix is enabled.
Reviewer Notes
You will need nix, a CAC, and access to staging.
Setup
The following commands should run successfully
rm -rf bin/prime-api-client
make bin/prime-api-client
bin/prime-api-client --cac --hostname api.stg.move.mil --port 443 list-moves

Code Review Verification Steps

[X] Request review from a member of a different team.

Warnings

:warning:
Please add the JIRA issue key to the PR title (e.g. MB-123)

  Generated by :no_entry_sign: dangerJS against 3ae902aea30b51e28c156252e264317abad2c97e


Considering using a more explicit variable name

In the code:
class UNet(object):
    def __init__(self, experiment_dir=None, experiment_id=0, batch_size=16, input_width=256, output_width=256,
                 generator_dim=64, discriminator_dim=64, L1_penalty=100, Lconst_penalty=15, Ltv_penalty=0.0,
                 embedding_num=1, embedding_dim=128, input_filters=3, output_filters=3):

self.input_filters = input_filters
self.output_filters = output_filters

I think changing input_filters and output_filters names to input_channels  and  output_channels  is more explicit.

I think whether channels or filters are different preferences by different naming conventions. This is inherited from https://github.com/yenchenlin/pix2pix-tensorflow repo, which I think it is better leave it for consistency.


The action 'Change Access Tier…' is enabled for blobs under Classic storage accounts

Storage Explorer Version: 1.15.0-dev
Build: 20200729.1
Branch: master
Platform/OS: Windows 10/ Linux Ubuntu 18.04
Architecture: ia32/x64
Regression From: Previous release (1.14.2)
Steps to reproduce:

Expand one Classic storage account -> Blob Containers.
Create a new blob container -> Upload one Block blob.
Check the blob's context menu.

Expect Experience:
The action 'Change Access Tier…' is disabled.

Actual Experience:
The action 'Change Access Tier…' is enabled.

More Info:

This issue also reproduces for one BlockBlobStorage account.

This is a side effect of enabling batch change access tier. I don't want to disable the action when the user have blobs that don't support access tier among their selection because currently there is no way to easily filter out those blobs in a select all scenario. The error detail is clear that access tier is not supported.

Hi @JasonYeMSFT Thank you for the update!

For blobs under Classic storage accounts. The error detail tells that access tier is not supported.
For blobs under BlockBlobStorage accounts. The error details like below.
<Message>Value for one of the query parameters specified in the request URI is invalid.\nRequestId:bf60c28c-501e-00a4-7e0a-66ceb6000000\nTime:2020-07-30T00:43:28.7464550Z</Message><QueryParameterName>comp</QueryParameterName>

We made a mistake by not saving classic account information for legacy quick access blob container nodes. Disabling this command by that will cause all quick access container nodes to lose this command. Move for re-triage.

Let's fix this in 1.15.0.

Hi @JasonYeMSFT
Verified on build 20200812.2. This issue has been fixed for the blobs. But it reproduces for the folders.

@v-limu, I am going to fix the block blob storage account scenario. For the classic storage account scenario, since the accountKind of blob containers in classic accounts are undefined, there is no way for us to distinguish them with the blob containers in Quick Access or local attachment that supports access tier. So that won't. be fixed.

Hi @JasonYeMSFT Got it. Thanks for your updates!

Verified on build 20200813.0. This issue has been fixed for one  BlockBlobStorage storage account. So close it.


Far future: experiment with auto-detection of form fields and add acroform automatically

I think a combination of pdfminer and reportlab could help us recognize fields about as well as Adobe Acrobat does and then add them as acroform.
Some places to start:
https://www.blog.pythonlibrary.org/2018/05/29/creating-interactive-pdf-forms-in-reportlab-with-python/
This suggests creating a separate PDF with the form elements and then merging. Not clear if reportlab can edit an existing PDF instead of merging. This looks quite plausible and not too much effort--the hardest part would be heuristics to identify form fields. Starting with looking for horizontal lines and boxes and then scanning for text near those boxes.
https://thecodework.com/blog/dynamically-changing-pdf-acroforms-with-python-and-javascript/ makes use of a package that lets you directly write to the Acrobat data stream. We would need to understand the low-level PDF format that makes up a form. This should definitely work but would be very high effort.

Recognizing boxes: https://medium.com/coinmonks/a-box-detection-algorithm-for-any-image-containing-boxes-756c15d7ed26

This should be a place for us to gather more research on this idea.

How would you recognize lists of fields that group the same information? E.g., 3 repeating form sections, asking for child 1, child 2, child 3.
How do we recognize tables with columns of data?

Some random research links:
https://medium.com/coinmonks/a-box-detection-algorithm-for-any-image-containing-boxes-756c15d7ed26
https://neptune.ai/blog/image-processing-in-python-algorithms-tools-and-methods-you-should-know
https://stackoverflow.com/questions/46617779/learning-method-for-detecting-relevant-fields-in-forms-image-format
https://github.com/OlafenwaMoses/ImageAI
https://hub.packtpub.com/opencv-detecting-edges-lines-shapes/

Random progress on this
Went through a lot of this today, writing down thoughts here.

Not clear if reportlab can edit an existing PDF instead of merging

reportlab seems to be one of those "open core" projects, where a lot of functionality is hidden behind a paid package. I also didn't see anywhere in that link that talked about merging PDFs, so I'm not super sure what you were talking about there.
pdfrw does seem to be promising, I couldn't get their examples working, but it seems promising, so I'm gonna look into more stuff tomorrow.
The box detection algorithms that opencv uses (particularly the coinmonks one) worked pretty well: on a screen shot of a random ME form, it gets the lines where fields are great:
This is the original form:

This is everything but horizontal and vertical lines removed.

Doesn't seem to catch:

different sections: the whole second part enclosed by a box kinda looks like a potential field. Could be remedied by OCRing text inside that box
check boxes (either too small or, in this case, looks too much like characters). Could be remedied with OCRing the text)

That was from a manual screen shot, but I'm sure of the hundreds of pdf libraries can render an image from just the file (maybe this blog post).

I had some more notes on Trello I'll copy here. Card link: https://trello.com/c/DUZ3qqGU
First: instead of pdfrw, it looks like pikepdf is the modern replacement. They have equivalent functionality.
Create an acroform with reportlab: https://thecodework.com/blog/dynamically-changing-pdf-acroforms-with-python-and-javascript/ (note this might be possible with pikepdf too, this just seems pretty nicely OO without requiring you to know PDF low level layout)
I found some libraries specifically for recognizing checkboxes that seems to work very well:  https://stackoverflow.com/questions/57260893/detect-horizontal-blank-lines-in-pdf-form-image-with-opencv
Line detection: https://stackoverflow.com/questions/57260893/detect-horizontal-blank-lines-in-pdf-form-image-with-opencv
The Trello card has some sample code I was playing with that worked decently. I think we would probably want to copy and combine the line detection with the box detection into one library instead of adding two dependencies that aren't frequently updated/actively maintained.

Darn, didn't realize you had been doing more work in the Trello :sweat_smile:  That definitely solves most of the issues, I think pdf2image, pikepdf (still wary about using reportlab), boxdetect, opencv, and maybe ocrmypdfor tesseract to get where text is (still looking through those options).

I wonder if the [ ] style checkbox will work with image recognition, or we should do a second pass to specifically scan for that common combination of characters to mean a checkbox field.

I now have a working prototype that adds AcroForm fields to a blank PDF and then merges that PDF, I went ahead and made a PR on the form explorer for now, but it's pretty loosely coupled, so could move anywhere.
Most of the effort was trying to understand low level PDF stuff, specifically what parts of the AcroForm (and the annotations) and such to copy over from a good PDF.
The next step is to wire up some existing opencv code that I have to recognize places where fields could be added a the PDF, and to pass it to the new tool.

Also want to link here: https://www.nonprofittechy.com/2022/01/05/batch-convert-adobe-livecycle-xfa-forms-into-ordinary-pdfs/ the notes on flattening XFA forms for future use.

Future development on this is happening in the FormFyxer https://github.com/SuffolkLITLab/FormFyxer/. But we can integrate the feature now--we're already integrating the field normalization/renaming functionality.


fix: remove "generate" ability from Baichuan-2-chat json config

Hello! Thank you for your contribution!
Can you please provide some context or explain the reasoning behind this change?


HTML example

I know this is a pretty dumb question, but I found so far the easiest way to get started with something is by having an example to read, e.g. for instance how to easily make something press able so I can use your extension and so on.
Again, this is probably very straightforward, but I haven't really programmed too much with js as I'm a researcher, but want to use your extension for some hierarchical visualization. So far cytoscape worked really great for this, but we need a way of managing the complexity.

@botev Expand-collapse is a major mechanism for managing complexity of your networks if you have a way to represent your relational information using nesting relations (i.e. compounds). For instance, if you have 100 high level nodes, each containing another 100 nodes as children (a total of 10K nodes), trying to visualize and analyze all this data is rather complex. However, if you display the graph in a format, where compound nodes are initially collapsed, then you only have 100 nodes in your graph. Later on, the user may expand these high level nodes on demand and focus on desired parts. Let us know if you have any questions about the usage of this extension for this purpose.

Yep I got it working. Sorry for some reason initially I missed that there is an demo.html. Must have been too late I guess.
I have only one, like a suggestion. Currently when you expand a node you have to redraw the graph based on the layout. It would be nice if it is possible to expand the node approximately at its location without rearranging the whole graph. Or maybe this is an issue only with the dagre layout.

You don't have to perform layout after expand-collapse; it's optional. See "layoutBy" option of this extension.

Hmm, to some extend what I have in mind is sort of on the lines:

When you expand you use the layoutBy only to arrange the nodes in the expanded node, while for instance anchoring its left top corner and all other nodes. After this any nodes outside nodes which turns out to be inside the area of the expanded node should be pushed away outside.

Atm, the expand makes redraw not only the expanded node, but the whole graph. Could you point me out to where is the code for redrawing the graph, I might be able to implement this myself and just add one more option to the extension.

Pushing neighbors away is done by "fisheye" option, which is already true by default. So all you have to do should be to set "layoutBy" null to avoid succeeding incremental layout. You shouldn't have to arrange the nodes inside the expanded node since presumably they were OK before they were collapsed. If you have to rearrange them however, you may do so by calling layout only on this child nodes (assuming your choice of layout support partial layout). Any particular reason you're using dagre? With compound nodes, I think you should definitely use "cose-bilkent" (not "cose").

Hmm for some reason atm if I set the "layoutBy" to null, collapse the nodes and then expand them I get all nodes on top of each other? Is there a flag for retaining their positions? I will also try "cose-bilkent", I tried "cose" but found "dagre" much better, but maybe your suggestion is even better.

@metincansiper will answer your question about laying out a part of a graph.
Regarding layout style to use, dagre seems to have done a very nice job in your particular example to my surprise since I thought it didn't handle compounds! I would be surprised to find out it handle arbitrary nesting levels and inter-graph edges though. With current cose-bilkent you cannot impose a hierarchy on your graph.

Yes, I think I've seen some quite bad renderings from dagre as well. I guess if there was something like a "tree" layout where you have some roots and layout the DAG according to levels based on the edges directions is what I need, but that might be a bit too much sideways from what I'm doing to write it on my own. I will keep in mind and explore both of the layouts and pick whichever on average seems to do better.
Anyway thanks a lot for the fast responses, was really helpful and I'm quite pleased with how easy and fast I managed to achieve what I wanted.

@botev you can set 'layoutBy' parameter as just layout options or whole layout function. I think that in your case it may be a function and you can use it in the following way:
layoutBy: function() {
// eles are the elements to layout
eles.layout(//your layout options here)
}
Not that this extension enables users to set the options after the initilazitaion as well. If you want to set this option after initilization you can use the following:
cy.setExpandCollapseOption('layoutBy', yourLayoutByFunction);

@metincansiper I think that did what I wanted. I have two questions:

How do I pass named arguments to the function ".layout()"? or where can I find the correct ordering of the parameters.
With this I get a very weird behaviour - when I press the collapse icon, the node collapses as desired. However it seems that the js does not recognize the mouserelease event, and essentially the node gets locked to the mouse movement (e.g. after the click until you make another click however you move the mouse the collapsed node follows the mouse movements). Can this be avoided?

So let me make it more clear in what I want to do:
Inside this function how do I get only the children of the expanded node, such that I specify the layout only for them and not modify anything outside this node.

Please refer to Cytoscape.js documentation!


Flexible Configuration Settings

Configuration Settings should be specified and enforced at Region/Zone/Appliance/Worker levels (currently at the Appliance Level only)

#7432 Configuration Revamp Phase 1 merged


Publish the teeracle build in the release for production

To be discussed

In the end, we'd like to offer signed enclave binaries, but once we'll have DCAP, we could place a special CI key in GA secrets to automate the signing without using our high-security key we registered with Intel


Explore validating the graph for each incoming event

Currently we validate the types within an event, and we enforce that certain fields in our legacy Tracker Protocol are correctly set, but we don't validate the event semantics as a whole.
To give some examples:

An iOS screen view lacking a geolocation context is a valid event
A PagerDuty event consisting of a single entity ("incident_raised") is a valid event
An event sent by a Snowplow Tracker without a verb is an invalid event
An ecommerce order without any line items is an invalid event

Relevant writing from Snowplow:

http://snowplowanalytics.com/blog/2013/08/12/towards-universal-event-analytics-building-an-event-grammar/
http://snowplowanalytics.com/blog/2014/03/11/building-an-event-grammar-understanding-context/
http://snowplowanalytics.com/blog/2015/01/18/modeling-events-through-entity-snapshotting/

This looks interesting: http://www.knewton.com/tech/blog/2012/02/graphlint-creating-a-domain-specific-language-for-graph-validation/

Migrated to https://github.com/snowplow/enrich/issues/146


503 Service unavailable for cgw/v1/chains call

Hi Team,
We're trying to setup safe via docker-compose steps mentioned in this repo.  for custom chain. FE is up  at https://safel2.sx.technology/. Calls to https://safel2.sx.technology/cgw/v1/chains fails with 503 Service unavailable.
Additional information:

Latest versions of all the services deployed.
Config service (CFG) is running and returns correct chainConfig at https://safel2.sx.technology/cfg/api/v1/chains/
Client Gateway (CGW) seems to be running fine and returns correct version at https://safel2.sx.technology/cgw/about
curl to CGW inside instance also returns 503. (Here 8000 is nginx port. No changes are made to nginx)

curl http://localhost:8000/cgw/v1/chains -L
{"code":503,"message":"Service unavailable"}

CFG debug logs when FE main page is rendered: (suspect 301 redirect might be an issue https://github.com/safe-global/safe-config-service/blob/main/nginx/templates/nginx.conf.template#L52 🤔)

 - - [18/Aug/2024:02:00:06 +0000] "GET /api/v1/chains?limit=20&offset=0 HTTP/1.0" 301 0 "-" "node"
2024-08-18 02:00:06,297 MT::GET::api/v1/chains/::35::200::/cfg/api/v1/chains/
 - - [18/Aug/2024:02:00:06 +0000] "GET /api/v1/chains/?limit=20&offset=0 HTTP/1.0" 200 1894 "-" "node"

CGW debug logs when FE main page is rendered:

{"build_number":"234a559","level":"error","message":{"chain_id":null,"client_ip":"xxx.xx.xx.xx","detail":"Service unavailable","method":"GET","origin":null,"path":"/v1/chains","response_time_ms":61.282322000712156,"route":"/v1/chains","safe_app_user_agent":null,"status_code":503},"request_id":"43837fda-36b7-410c-bc27-68edd468cd12","timestamp":"2024-08-18T02:00:06.304Z","version":"v1.53.0"}

DJANGO_ALLOWED_HOSTS & CSRF_TRUSTED_ORIGINS are set correctly (localhost:* , nginix:*, currendDomain allowed) in cfg.env. Same for txs.env

Hi team,
Any pointers? We are experiencing the same issue..


running elm-format failed

I installed elm-format globally via npm. Do I really need to install it locally for every project?
What is the best way to make this extension find it?
Thanks.

"This is just a simple extensions that execute elm-format if it exists on your global PATH. You will have to install elm-format and set the executable in your global PATH first before you can use this extension."
~/repos/elm-projs/PhotoGroove
18:37 $ elm-format
elm-format-0.18 0.6.0-alpha

Usage: elm-format [INPUT] [--output FILE] [--yes] [--validate] [--stdin]
                  [--elm-version VERSION] [--upgrade]

Looks like it is on my PATH!?
$ which elm-format
/usr/local/bin/elm-format

Indeed ;)
echo $PATH
...   :/usr/local/bin:  ...

So why won't VSC find it?

Ah, works now. There was a syntax error, oops!


Fix the speed of progress bar.

When the song is switched and the progress bar changes its position, it is quite slow.
So, the speed issue needs to be fixed.

Could you attach some SS?


Installing Python Module Error

When I try to install the python module, it throws an error.
pi@retropie:~/gunconf $ python setup.py install
Traceback (most recent call last):
File "setup.py", line 1, in
from setuptools import setup, find_packages
ImportError: No module named setuptools

sudo apt-get install python-setuptools

After installing python-setuptools with your command.  This is the next error I hit on installation:
pi@retropie:~/gunconf $ python setup.py install
running install
error: can't create or remove files in install directory
The following error occurred while trying to add or remove files in the
installation directory:
[Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/test-easy-install-1018.write-test'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
/usr/local/lib/python2.7/dist-packages/

Perhaps your account does not have write access to this directory?  If the
installation directory is a system-owned directory, you may need to sign in
as the administrator or "root" account.  If you do not have administrative
access to this machine, you may wish to choose a different installation
directory, preferably one that is listed in your PYTHONPATH environment
variable.
For information on other options, you may wish to consult the
documentation at:
https://pythonhosted.org/setuptools/easy_install.html
Please make the appropriate changes for your system and try again.

**sudo** python setup.py install

That did the trick. Thank you!