client: support custom client timeout (#2480)

What problem does this PR solve?
cherry-pick #2480.
What is changed and how it works?
This PR adds a new field in base client named timeout and it can be changed when we use WithCustomTimeoutOption to create a new client.
Check List

Tests

Unit test

Release note

Support setting a custom timeout for PD client

/merge

/merge


[BUG ]os-vmware pam related problem

I installed plugin os-vmware and wanted to execute command using vmWare tools, but I got back the error:
Failed to authenticate with the guest operating system using the supplied credentials.
I know that the credentials are correct.
In system.log I get this error:
vmsvc[86627]: in openpam_sidpatch(): all modules were unsuccessful for pam_sm_authenticate()
vmsvc[86627]: [critical] [vmsvc] Auth_AuthenticateUser:316:PAM failure - system error (4)
What I can do to solve this problem?

There is no such problem with plain FreeBSD 11, just now tested it.
OpnSense is the latest version - 17.1.1.

@maseron does it work if you enable "Disable integrated authentication" under System: Settings: Administration?

Thank You! That fixed the problem and saved my day :-)

Not so fast :D
We've switched 17.1 to integrated authentication. Do you have any idea which credentials VMware is using here? Do you use 2FA for the GUI authentication?

In my case VMware uses root user and password what I provide.
I do not use 2FA for GUI.

Thanks. It should still work if the passwords match for the GUI login. The system disables the root password, but PAM would still work.
Unless... https://github.com/opnsense/ports/blob/master/emulators/open-vm-tools/files/patch-scripts_linux_pam.d_vmtoolsd
VMware tries to authenticate against pam_unix, not system. oO

Hi @maseron,
I have a test package here that may work with and without integrated authentication. Would you mind trying it?
# pkg add -fA https://pkg.opnsense.org/snapshots/open-vm-tools-nox11-1280544_17,1.txz

You can change back to the old package by doing this:
# pkg install -Af open-vm-tools-nox11

Thanks,
Franco

@maseron on 17.1.2 we have a fix for enabled integrated authentication. would you mind trying it so this can be closed? thanks! :)

Hi, sorry for not replying. Upgraded to 17.1.2 and now customization works
also when the "Disable integrated authentication" checkbox is unchecked..
Thanks.
On Feb 26, 2017 11:39 AM, "Franco Fichtner"<EMAIL_ADDRESS>wrote:

@maseron https://github.com/maseron on 17.1.2 we have a fix for enabled
integrated authentication. would you mind trying it so this can be closed?
thanks! :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/opnsense/core/issues/1399#issuecomment-282544280, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ATKmDB2T-VtWkhvb-2INdMH5Akv2EZhOks5rgUhCgaJpZM4L_Af5
.

no worries. cool, I'll try to get this also into FreeBSD ports. Closing. :)


loss of image_inpainted_finally_checked

Why still miss files after downloading the latest dataset
(https://huggingface.co/datasets/haoningwu/StorySalon/blob/main/StorySalon.zip)
No such file or directory: './StorySalon/image_inpainted_finally_checked'
Could you provide the contents of all files in StorySalon

I would like to ask specifically how to place the data. For example, in StorySalon and testSet, I keep all the contents of StorySalon and put the video part of testSet into /StorySalon/image_inpainted_finally_checked, /StorySalon/mask, and /StorySalon/Text/Video respectively, but an error will be reported.

Sorry for the delayed response; I was busy with some other paper submissions.
We have already provided the data processing pipeline for StorySalon. Due to potential copyright concerns, we cannot directly provide all the video files. However, we do offer downloadable images from open-source libraries. You can follow the curation pipeline we provided to process and clean the data. The final processed images should be stored in the image_inpainted_finally_checked folder.
To facilitate reproducibility and simplify the data processing workflow, we have also included the index files (in txt/json format) of the processed data.


(#200) - Update index creation script to support vectors - Not create index

I'm using the version made available last Friday (08/25) that contains the option to create an index using a vector. Following the described step-by-step, I cannot create the index because it always displays the following message:
Chunking directories...
Total files to process=2 out of total directory size=2
Multiprocessing with njobs=4
0%| | 0/2 [00:00<?, ?it/s]SingletonFormRecognizerClient: Creating instance of Form recognizer per process
SingletonFormRecognizerClient: Skipping since credentials not provided. Assuming NO form recognizer extensions(like .pdf) in directory
SingletonFormRecognizerClient: Skipping since credentials not provided. Assuming NO form recognizer extensions(like .pdf) in directory
File (C:\project_documents\Documentation Master List.pdf) failed with 'object' object has no attribute 'begin_analyze_document
I also tried to create the index without using vector without informing the vector parameter, using the following command to create the index "az cognitiveservices account keys list --name "" --resource-group "< resource-group-name>" displays the following message:
Preparing data for index: gptkbindex
Traceback (most recent call last):
File "C:\projects\test\scripts\data_preparation.py", line 420, in
raise Exception("ERROR: Vector search is enabled in the config, but no embedding model endpoint and key were provided. Please provide these values or disable vector search.")
Exception: ERROR: Vector search is enabled in the config, but no embedding model endpoint and key were provided. Please provide these values or disable vector search.
In this case I am not using vector in the index and I believe it should be created without the vector option.

Hi @edercarlima we recently merged some fixes in #207 - do you mind pulling main and giving it another try? For the no-vector option, make sure config.json does not set vector_config_name otherwise it will still try to add vectors.

Hi @sarah-widder
I did the test without the vector_config_name parameter in the config.json file and I also couldn't create the index.
I'm going to download the version that contains fix 207 and test it.
Once the tests are finished, I'll let you know the result.

I was facing the same error as @edercarlima but, as @sarah-widder says, now is solved. Thanks!!

Hi @sarah-widder
I download the latest version from GitHub and managed to create the index without vectorization using the script. For this situation, in the config.json file I did not inform the parameter "vector_config_name": "default".
When trying to index with the vector option, include in the config.json file the parameter "vector_config_name": "default". and follow the step by step as described on github. When trying to create the index, it displays the following message:
Preparing data for index: gptkbindex02
Using existing search service gptkb-jyxxxxxxxxxx
Created search index gptkbindex02
Chunking directory...
Total files to process=4 out of total directory size=4
Multiprocessing with njobs=4
0%|                                                                                | 0/4 [00:00<?, ?it/s]SingletonFormRecognizerClient: Creating instance of Form recognizer per process
SingletonFormRecognizerClient: Skipping since credentials not provided. Assuming NO form recognizer extensions(like .pdf) in directory
File (C:\projetos_documentos\compliance\CO 0-00 - Lista Mestra da Documentação.pdf) failed with  'object' object has no attribute 'begin_analyze_document'
25%|████████████████████████████▎                                                                                | 1/4 [00:08<00:25,  8.39s/it]File (C:\projetos_documentos\compliance\CO 1-01 - Relação com Concorrentes e Política AntitrusteBS.pdf) failed with  'object' object has no attribute
'begin_analyze_document'
File (C:\projetos_documentos\compliance\CO 1-02 - Relação com FornecedoresBS.pdf) failed with  'object' object has no attribute 'begin_analyze_document'
SingletonFormRecognizerClient: Creating instance of Form recognizer per process
SingletonFormRecognizerClient: Skipping since credentials not provided. Assuming NO form recognizer extensions(like .pdf) in directory
File (C:\projetos_documentos\compliance\CO 0-00 - Manual de Documentação do Sistema de ComplianceBS.pdf) failed with  'object' object has no attribute 'begin_analyze_document'
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:09<00:00,  2.26s/it]
Traceback (most recent call last):
File "C:\projetos\sample-app-aoai-chatGPT-main\scripts\data_preparation.py", line 426, in
create_index(index_config, credential, form_recognizer_client, use_layout=args.form_rec_use_layout, njobs=args.njobs)
File "C:\projetos\sample-app-aoai-chatGPT-main\scripts\data_preparation.py", line 369, in create_index
raise Exception("No chunks found. Please check the data path and chunk size.")
Exception: No chunks found. Please check the data path and chunk size.
the data preparation script was assembled as follows:
python data_preparation.py --config config.json --embedding-model-endpoint https://gptkb-jyxxxxxxxxxx.openai.azure.com/openai/deployments/text-embedding-ada-002/embeddings?api-version=2023-06-01-preview --embedding-model-key 3b8888088d844c9b43488xxxxxxxxxx
Could you help me with this doubt?

Hi @edercarlima from the error message, it looks like you have PDF documents in your data folder, but you haven't passed through a Document Intelligence (Form Recognizer) resource and key to crack the PDFs. Please create a resource and use it to crack the PDFs https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/overview?view=doc-intel-3.1.0

Closing this issue.

I am running into similar issues with the latest version of this script. I can create an index without vectors, but not one with them. I did:
[
{
"data_path": "...",
"location": "...",
"subscription_id": "...",
"resource_group": "...",
"search_service_name": "...",
"index_name": "...",
"chunk_size": ...,
"token_overlap": ...,
"semantic_config_name": "default",
"language": "en",
"vector_config_name": "default"
}
]
With everything being the same as the config file for the non-vector index, except for the index name and adding "vector_config_name": "default".
I used the command:
py data_preparation.py --config config_vector.json --embedding-model-endpoint "https://.openai.azure.com/openai/deployments//embeddings?api-version=2023-11-01" --embedding-model-key
I get the error message:
Data preparation script started Preparing data for index: <index name> Using existing search service <search service name> Created search index <search index name> Chunking path <path>... Total files to process=1 out of total directory size=1 Multiprocessing with njobs=4   0%|                                                                                | 0/1 [00:00<?, ?it/s]SingletonFormRecognizerClient: Creating instance of Form recognizer per process SingletonFormRecognizerClient: Skipping since credentials not provided. Assuming NO form recognizer extensions(like .pdf) in directory 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [04:00<00:00, 240.70s/it] Traceback (most recent call last):   File "<preceeding path>\sample-app-aoai-chatGPT\scripts\data_preparation.py", line 464, in <module>     create_index(index_config, credential, form_recognizer_client, embedding_model_endpoint=args.embedding_model_endpoint, use_layout=args.form_rec_use_layout, njobs=args.njobs)   File "C<preceeding path>\sample-app-aoai-chatGPT\scripts\data_preparation.py", line 404, in create_index     raise Exception("No chunks found. Please check the data path and chunk size.") Exception: No chunks found. Please check the data path and chunk size.
What is going on?
What deployment endpoint am I supposed to be using? I cannot see one specifically for the embedding deployment (and not the whole service) listed on the Azure portal or on Azure OpenAI studio. I tried both api-version=2023-06-01-preview and api-version=2023-11-01. Neither worked, and in both cases, an empty index using an old api version was used (2023-07-01-preview).
What am I supposed to do about the embedding key? I cannot find one specifically for the embedding deployment. I tried using the key for my Azure OpenAI service as a whole, but including and excluding it did not seem to make a difference. In fact, I could not see where it was used in the code (which I do not fully understand).
Why does this successfully chunk my file and create an index when I am not using vectors, but fail when I am trying to use them?

I have exactly the same issue as @jbao8899. Have you manage the way to solve this?


chore: modernize templates

Modernize old templates:

Fauna Boilerplate
Mongo
Upstash Geolocation
Uptastash Rate Limit
Hello World
GH Auto Deploy
QStash

Can we used a new versions of actions?
actions/checkout@v4
actions/setup-node@v4


microsot > microsoft

Article Latest updates and version history for SQL Server has a link to aka.ms/sqlserverbuilds where in sheet 2017 of the Excel file there is a typo in link KB 5010786 which now leads to https://support.microsot.com/help/5010786.
microsot > microsoft
Click to download this new Excel file now

@ingebeumer Thanks for pointing out this issue. I will share this to the owner of the Excel file.

The issue has been fixed.


Add the pick list for the build presets

The portal has been updated to allow the user to select the build presets. These populate the app locaiton, api location, and the app artifact location. see below for details

cc @fiveisprime as per our conversation

Did a little bit of investigating and these values are all hard-coded right now.  The team is using this table as a reference: https://docs.microsoft.com/en-us/azure/static-web-apps/front-end-frameworks
We can do the same, maybe have a learn more link directing them to the table.  If they hit Custom, then they'll get the 3 prompts as usual, otherwise we'll use the presets and skip the prompts.

cool.
This will evolve, I am sure. But I think this is a great step forward.


paictl pai build

Build hadoop-ai or download form  official hadoop, and then build all service image:

sudo ./paictl.py image -p /path/to/cluster-configuration/folder -a build

Push service image to the docker registry in the cluster configuration

sudo ./paiclt.py image -p /path/to/cluster-configuration/foler -a push

push and build

sudo ./paictl.py image -p /path/to/cluster-configuration/foldler

Some change:

Service.yaml's image list will be replace by the file image.yaml in each image's folder.
paiBuild lib
Some handler in the common lib.

Coverage increased (+13.6%) to 67.407% when pulling b6f1f6f2680dcf4f44d662ab60c3e1f014ad6fdd on yuye/paictl-paiBuild into 78668766d4674c785bc98f74688d38197547da54 on management-dev.

Coverage increased (+13.6%) to 67.407% when pulling b6f1f6f2680dcf4f44d662ab60c3e1f014ad6fdd on yuye/paictl-paiBuild into 78668766d4674c785bc98f74688d38197547da54 on management-dev.


Remove extraneous switch in QuickAlchemy.js

Small typo here was preventing the file from loading.

Thanks for catching this! I have pushed the fix through.


iOS Safari bug when toggling between multiple states

I have a click handler that toggles on and off multiple zoom behaviors.
clickHandler(){
  if (state1Selected){
      d3CanvasSelector.call(zoomBehavior2).on('.zoom .start .end', null)
      d3CanvasSelector.call(zoomBehavior1);
  } else if ( state2Selected) {
      d3CanvasSelector.call(zoomBehavior1).on('.zoom .start .end', null)
      d3CanvasSelector.call(zoomBehavior2);
   } else {
       d3CanvasSelector.on('.zoom .start .end', null)
   }
}

It works as expected on chrome and safari. But it ends up breaking in iOS safari after this sequence of events.
state1Selected - drag and zoom work fine
state2Selected - drag and zoom work fine
state1Selected - when trying to drag, transform.k is getting increased/decreased. behavior breaks down.
Will try to replicate in a open source environment when I get the time.

Closing to inactivity.


baseport for content cluster nodes doesn't work well for all node types

Describe the bug
When we set baseport for content cluster nodes, it works for storagenode but doesn't work well for distributor/searchnode.
distributor-base-port also doesn't work at all.
To Reproduce
Steps to reproduce the behavior:

Deploy one content cluster with baseport

  <content id="test2" version="1.0">
    <redundancy>1</redundancy>
    <documents>
      <document type="test2" mode="index" />
    </documents>
    <nodes>
      <node hostalias="node1" distribution-key="0" baseport="20000"/>
    </nodes>
  </content>

Add another content cluster before the existing cluster

  <content id="test1" version="1.0">
    <redundancy>1</redundancy>
    <documents>
      <document type="test1" mode="index" />
    </documents>
    <nodes>
      <node hostalias="node1" distribution-key="0" baseport="10000"/>
    </nodes>
  </content>

  <content id="test2" version="1.0">
    <redundancy>1</redundancy>
    <documents>
      <document type="test2" mode="index" />
    </documents>
    <nodes>
      <node hostalias="node1" distribution-key="0" baseport="20000"/>
    </nodes>
  </content>

See warnings and restart messages

WARNING distributor cannot reserve port 10000 on vespa-container: Already reserved for storagenode. Using default port range from 19109
WARNING distributor cannot reserve port 10001 on vespa-container: Already reserved for storagenode. Using default port range from 19110
WARNING distributor cannot reserve port 10002 on vespa-container: Already reserved for storagenode. Using default port range from 19111
WARNING distributor2 cannot reserve port 20000 on vespa-container: Already reserved for storagenode2. Using default port range from 19120
WARNING distributor2 cannot reserve port 20001 on vespa-container: Already reserved for storagenode2. Using default port range from 19121
WARNING distributor2 cannot reserve port 20002 on vespa-container: Already reserved for storagenode2. Using default port range from 19122
WARNING Change(s) between active and new application that require restart:
In cluster 'test2' of type 'content':
    Restart services of type 'distributor' because:
        1) stor-communicationmanager.mbusport has changed from 19109 to 19120
stor-communicationmanager.rpcport has changed from 19110 to 19121
stor-status.httpport has changed from 19111 to 19122
In cluster 'test2' of type 'search':
    Restart services of type 'searchnode' because:
        1) # Port to use for the rpcserver.
proton.rpcport has changed from 19103 to 19114
# Port to use for the web server
proton.httpport has changed from 19107 to 19118
# Connect spec for transactionlog server.
# TODO Consider not using RPC at all
proton.tlsspec has changed from "tcp/vespa-container:19108" to "tcp/vespa-container:19119"
# Port number to use for listening.
translogserver.listenport has changed from 19108 to 19119

Expected behavior
baseport setting should work properly for distributor/searchnode and we should be able to avoid restarting services.
Environment (please complete the following information):

OS: Ubuntu
Infrastructure: self-hosted (observed the same behavior on Kubernetes)
Versions 18.04

Vespa version
8.339.15
Additional context
We want to avoid restarting unrelated distributor/storagenode/searchnode when adding/removing/updating content clusters because it causes outage for a while.
baseport setting works for storagenode but doesn't work well for distributor (distributor-base-port also doesn't work) and doesn't affect searchnode.

After reproducing this locally I can confirm your observations that distributor-base-port simply doesn't work at all. I believe this particular property is a remnant from an older version of the content cluster model, which probably shouldn't have been documented at all. It belongs in a museum! 🦖
You are also correct that the searchnode service does not appear to get its ports assigned as expected relative to the specified base port. I'm not sure of the reason behind this, but a lot of the port assignment logic has been refactored over the years. Since we run our nodes in separate containers (without specifying base ports) instead of co-located, this means we probably haven't noticed any regressions popping up here.
I'd strongly suggest using containers to avoid this problem altogether. This is where our development focus is concentrated and deployment friction can be expected to be minimal. Unless there are technical reasons that preclude using containers for this use case?

Thank you for taking care of this.
We basically followed the sample setting (e.g. https://github.com/vespa-engine/sample-apps/blob/master/examples/operations/multinode-HA/services.xml).
What do you mean by containers and how can we set up?

I see, we are already using Kubernetes pods.
We have many content clusters and each of them uses the grouped distribution with multiple nodes.
If we assign different nodes for them, we need to manage hundreds of nodes and it's not realistic.
So we want to use the same nodes in different content clusters.

we need to manage hundreds of nodes

Really, the right solution here is to have Vespa Cloud manage it, which it can do even though they are in your account. I'm forgetting why you said this wasn't an option for you, but if it might help speaking to the right people on your side we can try to do that for you. You can mail me, bratseth at vespa.ai.

Another member in our team is already talking about Vespa Cloud.
Ultimately we want to use it but we cannot do it soon.
It's great if there is a solution to handle this issue through configurations.

We see that solution as inferior to using multiple containers, since this provides uniform management of nodes and resource isolation between clusters, so we are unlikely to spend any time on this.


Removed parameter 'b' from method 'calculateSomething(int, int, int)'

Hi, I'm a refactoring bot. I found and fixed some code smells for you.
You can instruct me to perform changes on this pull request by creating line specific comments inside the 'Files changed' tab of this pull request. Use the english language to give me instructions and do not forget to tag me (using @) inside the comment to let me know that you are talking to me.

@Refactoringbot-Test Could you rename this method to "testSomething(int, int, int)" please?


Document the behavior of R_RISCV_ALIGN

Describe the behavior of R_RISCV_ALIGN relocation.

@MaskRay

Changes:

Apply @jim-wilson's and @MaskRay's suggestion.
Rebase to master to fit new structure of document.

Changes:

Minor fix typo.

So, what happens with things like:
.byte 0
.balign 16

Is that currently handled correctly? What does it even emit? We need to make sure all edge cases are covered before we commit to this being an official part of the ABI, and I suspect currently things break if you do something like that.

Currently binutils will result linking error, but I think it should padding with a NULL byte at first and then 14 byte NOPs, I've add the behavior into spec.

@jim-wilson Could you review the updated version :)
@Nelson1225 Could you review that to make sure that is consist with binutils except the handling for 1-byte alignment (.byte 0 + .balign 16).

So, what happens with things like:
.byte 0
.balign 16

Is that currently handled correctly? What does it even emit? We need to make sure all edge cases are covered before we commit to this being an official part of the ABI, and I suspect currently things break if you do something like that.

Currently binutils will result linking error, but I think it should padding with a NULL byte at first and then 14 byte NOPs, I've add the behavior into spec.

What I expected is that we probably need to make sure the boundary (from data to text) align to the text alignment.  And we now have mapping symbols in GNU binutils (may need llvm guys to make sure if it is also supported in llvm), so it shouldn't be hard to find the boundaries that need extra alignment padding.  For example,
.byte 0 -> data, $d
.balign16 -> fill nops in text section, so this is part of text, $x
If the rvc is enabled/disabled (text alignment is 2/4 byte), then we can do the 2/4-byte alignment after the `.byte 0' first, then the following alignments should work as expected.  And we probably need to do the same thing for the boundary from rvc to norvc, and probably also need to consider the .option relax/norelax.  Ideally, we can handle these boundaries as the above idea, but not sure if it can work for every cases.  Considering data/text, rvc/norvc and relax/norelax at the same time makes things complicated.  However, I have discussed this with Palmer and Kito many times, and Palmer also discussed this with Jim and Andrew before, maybe we should continue to recommend users not to put their data in text until we have a good solution.

Currently binutils will result linking error, but I think it should padding with a NULL byte at first and then 14 byte NOPs, I've add the behavior into spec.

Yes, the NULL byte padding is what I expected - align the boundary from data to text, if rvc is enable, then we need 1 byte padding, and 14 bytes NOPS. Otherwise we need 3-byte padding, and 12 bytes NOPS.

1 byte padding + 14 bytes NOPS or 3-byte padding + 12 bytes NOPS are not too much difference on no-RVC case, since both are not valid instruction sequence if CPU not support RVC, I think we could using some more general term like padding byte instead of NULL byte here to let implementation has flexibility.

LGTM, thanks.

Changes:

Revise the description for the padding bytes.

So, what happens with things like:
.byte 0
.balign 16

Is that currently handled correctly? What does it even emit? We need to make sure all edge cases are covered before we commit to this being an official part of the ABI, and I suspect currently things break if you do something like that.

Currently binutils will result linking error, but I think it should padding with a NULL byte at first and then 14 byte NOPs, I've add the behavior into spec.

What I expected is that we probably need to make sure the boundary (from data to text) align to the text alignment. And we now have mapping symbols in GNU binutils (may need llvm guys to make sure if it is also supported in llvm), so it shouldn't be hard to find the boundaries that need extra alignment padding. For example,

Uh, I hope you're potentially willing to change the implementation, because that pull request has not been merged yet. I don't foresee it changing, but it is not a finalised spec.

ping

Changes:

Rebase

@rui314 @MaskRay I would like to gather LGTM from you guys, since we already got two LGTM from GNU developer but no LGTM from non-GNU developer  :)

The relocation type R_RISCV_ALIGN marks a location that must be aligned to N-bytes

To me, this sounded like R_RISCV_ALIGN appears past-the-end of the padding nop instructions, as that's the location that will be aligned.

Changes:

Add example to show how R_RISCV_ALIGN is used.
Add pseudocode to show how to compute alignment of R_RISCV_ALIGN.
Add refer to IALIGN when it is first mentioned.
Add more word to clarify the location of R_RISCV_ALIGN is point to beginning of padding bytes.

Ping :)

What about my previous comment?

To me, this sounded like R_RISCV_ALIGN appears past-the-end of the padding nop instructions, as that's the location that will be aligned. Maybe we should make it more clear that the relocation refers the beginning of the nops?

What about my previous comment?

Does this clear or its still not clear enough? https://github.com/riscv-non-isa/riscv-elf-psabi-doc/pull/210/commits/cb8a12e99efb618f1ab82dfe1cca2e6b6db14621
 The relocation type `R_RISCV_ALIGN` marks a location that must be aligned to
 `N`-bytes, where `N` is the smallest power of two that is greater than the value
 of the addend field, e.g. `R_RISCV_ALIGN` with addend value 2 means align to 4
-bytes; this relocation is only required if the containing section has any
-`R_RISCV_RELAX` relocations.
+bytes, `R_RISCV_ALIGN` with addend value 4 means align to 8 bytes; this
+relocation is only required if the containing section has any `R_RISCV_RELAX`
+relocations, `R_RISCV_ALIGN` points to the beginning of the padding bytes,
+and the instruction that actually needs to be aligned is located at the point
+of `R_RISCV_ALIGN` plus its addend.

LGTM
nit: there's a typo: diretive → directive

So, what happens with things like:
.byte 0
.balign 16

Is that currently handled correctly? What does it even emit? We need to make sure all edge cases are covered before we commit to this being an official part of the ABI, and I suspect currently things break if you do something like that.

Currently binutils will result linking error, but I think it should padding with a NULL byte at first and then 14 byte NOPs, I've add the behavior into spec.

What I expected is that we probably need to make sure the boundary (from data to text) align to the text alignment. And we now have mapping symbols in GNU binutils (may need llvm guys to make sure if it is also supported in llvm), so it shouldn't be hard to find the boundaries that need extra alignment padding. For example,
.byte 0 -> data, $d .balign16 -> fill nops in text section, so this is part of text, $x
If the rvc is enabled/disabled (text alignment is 2/4 byte), then we can do the 2/4-byte alignment after the `.byte 0' first, then the following alignments should work as expected. And we probably need to do the same thing for the boundary from rvc to norvc, and probably also need to consider the .option relax/norelax. Ideally, we can handle these boundaries as the above idea, but not sure if it can work for every cases. Considering data/text, rvc/norvc and relax/norelax at the same time makes things complicated. However, I have discussed this with Palmer and Kito many times, and Palmer also discussed this with Jim and Andrew before, maybe we should continue to recommend users not to put their data in text until we have a good solution.

Just mentioned that there is a patch from Jan Beulich, who is one of the binutils global maintainers, seems trying to resolve the above rvc to norvc alignment issue, here.


refresh and fetch description and profile image

Changes:

Changed contract dedupe to check for all metadata fields across all contracts received
Added collection image url support to opensea and objkt providers
Added script for refreshing contracts

this is fantastic, benny! super pumped!
@radazen any concern with writing the CreatorAddress on the contract? guessing no because this is within a separate Descriptors field

Shouldn't be a problem! We already map the contract's CreatorAddress to the database's owner_address field; this just does a better job processing that data before we write it.

Closing remark for this PR, we should probably run the profile images through the media pipeline at some point instead of using opensea's profile images directly.


Would it be possible to mark Event.eventTime as Date??

Okay, I admit I'm asking for this feature here because upstream kubernetes has yet to merge https://github.com/kubernetes/kubernetes/pull/100600 and/or resolve https://github.com/kubernetes/kubernetes/issues/82665, so please feel free to close this if you want to follow only spec (would totally understand that!).
In an Event, the eventTime is required/not allowed to be nil:
https://github.com/swiftkube/model/blob/2d10b91ee6fa3584807ec859370b62512760c4ec/Sources/Model/events/v1/Event%2Bevents.v1.swift#L74
Which correctly follows the spec provided by upstream.
That being said, because kubelets aren't yet using the Events v1 API, eventTime is sometimes nil in the response from the cluster:
{
  "kind": "EventList",
  "apiVersion": "events.k8s.io/v1",
  "metadata": { "resourceVersion": "10891762" },
  "items": [
    {
      "metadata": { ... },
      "eventTime": null,
      "reason": "FreeDiskSpaceFailed",
      "regarding": {
        "kind": "Node",
        "name": "k3s-node-1",
        "uid": "k3s-node-1"
      },
      "note": "(combined from similar events): Failed to garbage collect required amount of images. Attempted to free 462039449 bytes, but only found 0 bytes eligible to free.",
      "type": "Warning",
      "deprecatedSource": { "component": "kubelet", "host": "k3s-node-1" },
      "deprecatedFirstTimestamp": "2024-01-27T06:54:50Z",
      "deprecatedLastTimestamp": "2024-02-10T19:15:03Z",
      "deprecatedCount": 3679
    }
  ]
}

I've truncated .items[0].metadata because it wasn't relevant.
Would it be possible to type Event.eventTime as Date?, allowing the JSONDecoder to decode these events as well?
I discovered this when I couldn't get any events from the kubelet/node:
(lldb) p try jsonDecoder.decode(SwiftkubeModel.events.v1.EventList.self, from: data)
(DecodingError) <invalid> (0x80)

where data is NSData containing the full reply from the server.
I compared the server response against the struct types and found that discrepancy. Tested isolating it to just that struct member/type and got the same error:
(lldb) p try jsonDecoder.decode(Date.self, from: Data(base64Encoded: "bnVsbA==")!)
(DecodingError) <invalid> (0x80)

edit: FWIW, I found https://github.com/swiftkube/model/blob/2d10b91ee6fa3584807ec859370b62512760c4ec/Sources/Model/core/v1/Event%2Bcore.v1.swift#L62 which has different keys, so I don't think this is the right struct to use?

Sorry, nevermind, user error - I realized there's also the core.v1.Event accessed via client.events instead of client.eventsV1.events, which would be sufficient until upstream migrates everything to their Events V1

Hey there, sorry for not replying sooner. I was a little bit under the weather the last week.
Glad it worked out with core.v1.Event 😸


issue #3982: fix setup cygwin action.

fix setup Cygwin action by update to its lastest commit.

This PR try to fix #3982 .
Root Cause
According to the https://github.com/cygwin/cygwin-install-action/blob/master/action.yml
cygwin-install-action will check the the signature by default.
But http://mirrors.kernel.org/sourceware/cygwin/x86_64/  moved the signatures to single file yesterday.
http://mirrors.kernel.org/sourceware/cygwin/x86_64/sha512.sum
And https://cygwin.com/setup-x86_64.exe didn't know how to check the signatures in new place.
Solutions
disable the signatures check temporary. I also update the cygwin-install-action to latest commit(this may be unnecessary) .
But I believe the cygwin will collaborate with kernel.org to fix this bug soon, then we can open the signature check again later.

Merged to #3973


where to get pretrained mb2-ssd-lite weight?

Hi, I want to train a detection model, and in your doc/pytorch-ssd.md, only mb-v1 pretrained weight mentioned, as follows:
# you only need to run these if you aren't using the container
$ cd jetson-inference/python/training/detection/ssd
$ wget https://nvidia.box.com/shared/static/djf5w54rjvpqocsiztzaandq1m3avr7c.pth -O models/mobilenet-v1-ssd-mp-0_675.pth
$ pip3 install -v -r requirements.txt

I want to use a different net, for example mb2-ssd-lite, where to get the corresponding pretrained weights?  In the model downloading page you provided, the model archive seems not exactly same as the mobilenet-v1-ssd-mp-0_675.pth, which has a .uff suffix, like ssd_mobilenet_v1_coco.uff. They are used for inferencing only and not for training? Badly confused.

@CapBarbossa the other pretrained models available can be found in the upstream repo at https://github.com/qfgaohao/pytorch-ssd#download-models
I believe the mb2-ssd worked with ONNX export/import into TRT, but wasn't noticeably faster


Minor cleanup of MetaDEx status and RPC output

This PR performs the following:

Switches away from string based statuses in code

With the amountremaining and amounttofill MetaDEx fields;

Moves them out of omni_gettransaction and into omni_gettrade
Hides the fields if the trade is not open/active

Fixes a bug where status would show as cancelled if the trade was invalidated during parsing, eg:

   "status" : "cancelled",
   "canceltxid" : "0000000000000000000000000000000000000000000000000000000000000000",
   "matches" : [
   ],
   "valid" : false,

For further info please see discussion in https://github.com/OmniLayer/omnicore/issues/200

Fixes a bug where status would show as cancelled if the trade was invalidated during parsing

Good catch, I noticed this too and handwaived it. I like your new approach much more though! :)
I'll give it a test soon, but it looks good to me.
FYI: you can close open issues by mentioning it in the PR message.

I'm split regarding the behavior of filled or cancelled orders and "amountremaining", "amounttofill".
I can see why you exclude them for closed orders, so I just want to name some alternatives:

we could also set those values to zero, if the trade is no longer active (this may be easier for interacting with the API, because the schema for a trade is always the same)
we may use the fields as hint, to show how many units were sold in total (it's slightly redundant though, given that the total can be calculated, and it doesn't make much sense, because there are no tokens left for sale)

I'm wondering, if we should rename "amountforsale" and "amountdesired" and somehow highlight that these values refer to the "initial" or "original" amounts.
Anyway, those points are just for further consideration and brainstorming, and the PR is good! :)

we could also set those values to zero, if the trade is no longer active (this may be easier for interacting with the API, because the schema for a trade is always the same)

I'm open to this if that helps maintain a static schema, though I'm a little concerned people may use zero as a literal value for closed trades so ideally if we go down this path we would make sure amountremaining and amounttofill don't need to be used to infer other values.

we may use the fields as hint, to show how many units were sold in total (it's slightly redundant though, given that the total can be calculated, and it doesn't make much sense, because there are no tokens left for sale)

I'd rather be explicit, and if we want to provide this data then we should provide amountsold and amountreceived fields

I'm wondering, if we should rename "amountforsale" and "amountdesired" and somehow highlight that these values refer to the "initial" or "original" amounts.

I see...  Any suggestions? ````originalamountforsaleI think is going to raise more questions than it answers hehe...  I'm not sure I see much of an issue (eg it seems fairly straight forward thatamountsold=amountforsaleminusamountremaining``` etc)


Review microcontroller lesson and make improvements

We should make some updates to this before moving forward this year. Some ideas (be sure to sure the list for more):

We have increased the time from 1 hr to up to 3hrs in the lab portion so we can expand a bit or just give students more time to complete their microcontroller.
Can we improve how the setup works interm of how to heat the sensor? Something more realistic?
The lecture portion that went over coding was too little time to cover the material.
We should get them to actually turn in the homework assignment for a grade this time.

See this issue: https://github.com/moorepants/eme185/issues/14

I came up with a pretty neat idea, though it involves a pretty serious overhaul of the lesson.
Wire up a photocell in a voltage divider to an analog input pin, then put an LED on a PWM output. They can be set up on a breadboard so the LED can meaningfully influence the photocell's reading. You can take an ambient lighting measurement, then choose a setpoint a bit brigher than that. Then you write a controller to turn the LED brightness up or down to reach the set brightness level. Once it's running, you can move your hands around to create shadows to perturb the input and watch the LED get brighter to compensate. I tested it out and it works surprisingly well.
However, I just found that last year's BOM for getting the lesson materials includes components to implement this idea. Was there a big reason not to? It has quite a few advantages over the temperature controller: it's much more responsive to perturbations, it's visually appealing (can see the LED dim and brighten), and it's a simpler physical setup.

This sounds quite nice. I don't recall what we discussed last year and why we choose what we did. Is the controller just a proportional controller? Something the can easily implement in a few lines of code?
What BOM are you talking about?
I'm willing to go with this. Can you give me an idea of how much it would cost for new hardware?
Also, are you (and other TAs) willing to redo the lab portion (mainly the slides)?

I spent very little time playing with controller gains, but a proportional controller can work but gives steady state error (kind of cool to demo that I think), and an integral term can get rid of it.
The folder from last year with the Arduino lessons has a pricing spreadsheet with costs of components.
~$70 for 50 of everything required (aside from what we already have).
I'm willing to do it because I can't really get the temperature controller to work well at all.

Here's a screenshot to show it in action. It's just the 10-bit reading from the sensor subtracted from 1023 to turn it into a "brightness" value. I'll mess around with it a little more to see if it can be a little cleaner (I think Kp is a bit high maybe):

It is better to give a detail example, i.e. LED brightness control during the lecture, include the controller design, program design. After the lecture, students can know how to writer code to read the signal and set the pins of the micro controller.
The example will act as a template for the student, thus they just need to add function to the example code for their own project. In that case, DAC, ADC, PWM, IO read/write, while/if else  program should be included.

This has been settled on. Tutorial can be found here


Bug: using &macro_param_1to1 for the second time in a parameterized macro fails to include the second MACRO_PLACEHOLDER

How to reproduce:

Add a new macro. It will start with a &none
Add a &macro_param_1to1
Move the &none to the end of the list, change it to &kp
Add a 2nd &macro_param_1to1
Add a binding (default is &kp)

And this is what's produced from View keymap data:
        macro_test: macro_test {
            compatible = "zmk,behavior-macro-one-param";
            #binding-cells = <1>;
            bindings = <&macro_param_1to1 &kp MACRO_PLACEHOLDER &macro_param_1to1 &kp>;
            label = "MACRO_TEST";
        };

which is missing the second MACRO_PLACEHOLDER at the end of the bindings = line.

Ah, that's not great. I'll have to get back to take a deeper look later but I can reproduce it easily so I don't expect it'll be too much trouble. Thanks so much for the heads up!

So the simplest summary is that it only injects the MACRO_PLACEHOLDER parameter on the first instance of any single &macro_parameter_XtoY directive. The oversight is caused by reusing the same code I to determine the parameter types. In that context I only want to consider the first instance, but it's still valid to reuse one parameter multiple times.

Thanks! Multiple &macro_parameter_1to1 are used for example if you want to parametrize both the &macro_press and &macro_release (with e.g. a mod key)
While you are at it, there is another bug that may be related: if you first add two (non-parametric) bindings (e.g., &kp A and &kp B), then add two &macro_parameter_1to1, then move &kp A and &kp B each after their respective &macro_parameter_1to1, then the code generated is
        new_macro: new_macro {
            compatible = "zmk,behavior-macro-one-param";
            #binding-cells = <1>;
            bindings = <&macro_param_1to1 &kp MACRO_PLACEHOLDER &macro_param_1to1 &kp B>;
            label = "NEW_MACRO";
        };

where the trailing B in the bindings = line should not have been there.

Yeah, so there's two pieces I think:

Changing a &none binding to &kp will normally require you to add the keycode parameter, except in the context of a macro where it's parameterized in which case I think (haven't studied that code for a while now) it'll inject it for you, but adding a &kp binding through the Add Key Binding button will add it to the array without enforcing the same restriction or applying the same side effects
When serializing macros it deals with convenience bindings like RGB_COLOR_HSB(...) and injects MACRO_PLACEHOLDERs where appropriate, except in this case where its the second instance of that parameter being applied.

The other issue you've raised is a result of point number 2, where it's being overlooked because that macro parameter has been dealt with already. It's not the end of the world, because the actual value is meaningless, but it's inconsistent/unexpected and it's indicative of the larger issue that the logic isn't applying in all cases.
I'll need to look at how I can do this cleanly without breaking the parameter type resolution as it exists or duplicating parts of it but it shouldn't be that bad. If I have some quiet time tomorrow evening that should be enough.

Sorry, I haven't forgotten about this I just haven't been able to concentrate on solving it. I have the code changes in place but I want to write a quality test or two before deploying anything.


Chinese SubwordTextEncoder alphabet can't find english subwords

I'm training on an English-Chinese dataset that sometimes has proper nouns in English:
source: "45.00 kilometer(s) from StaBfurt"
target: "距 StaBfurt 45.00 公里"
I've increased vocab size to 2**17 and can see that target_vocab.vocab_size and len(target_vocab._alphabet) have maxed out at ~71k and 3.9k respectively.
however, i get a crash when during bi_vocabs_token_generator at  target_vocab.encode(target.strip()).
I can reproduce the assertion fail at the word StaBfurt (see below).
The Chinese vocab was generated and should have included this word.
Is there something I should be doing here?
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-40-d8eafb9371e8> in <module>()
----> 1 target_vocab.encode("StaBfurt")

//anaconda/lib/python3.5/site-packages/tensor2tensor/data_generators/text_encoder.py in encode(self, raw_text)
    403     """
    404     return self._tokens_to_subtoken_ids(
--> 405         tokenizer.encode(native_to_unicode(raw_text)))
    406
    407   def decode(self, subtokens):

//anaconda/lib/python3.5/site-packages/tensor2tensor/data_generators/text_encoder.py in _tokens_to_subtoken_ids(self, tokens)
    433       ret.extend(
    434           self._escaped_token_to_subtoken_ids(
--> 435               _escape_token(token, self._alphabet)))
    436     return ret
    437

//anaconda/lib/python3.5/site-packages/tensor2tensor/data_generators/text_encoder.py in _escaped_token_to_subtoken_ids(self, escaped_token)
    495     return [
    496         self._subtoken_string_to_id[subtoken]
--> 497         for subtoken in self._escaped_token_to_subtoken_strings(escaped_token)
    498     ]
    499

//anaconda/lib/python3.5/site-packages/tensor2tensor/data_generators/text_encoder.py in _escaped_token_to_subtoken_strings(self, escaped_token)
    481         # characters in the token is not in the alphabet. This should be
    482         # impossible and would be indicative of a bug.
--> 483         assert False, "Token substring not found in subtoken vocabulary."
    484
    485     return ret

AssertionError: Token substring not found in subtoken vocabulary.

(edited for more complete explanation)

I also encounter this problem, delete the line:

#assert False, "Token substring not found in subtoken vocabulary."
and add the code:

break
seems fine.

closed -- this issue goes away with using a chinese word tokenizer to segment words before running SubwordTextEncoder


Add Artifact.Deleted flag

We need to know if an artifact was deleted. Add this field.
See workflow_types.go in #8530.

@alexec I see you added the field to ArtifactLocation rather than to Artifact directly. I am assuming I should do the same (although by the title of this issue it's not clear)?

I'm not 100% sure, I think it is Artifact. If that does not work, it must be ArtifactLocation.

There's an "ArtifactLocation" for the Workflow itself as well as one for each Artifact which can override the default.  But I am thinking of putting it in Artifact since it represents the state of the Artifact itself.


Using multiple "files:" entries in config causes bootloops

Description
One entry of
   storage:
      files:
        - path:

works.
Adding a second files/path entry causes a bootloop.
Impact
Major blocker - cannot build working Flatcar VMs that require more than one file be included in the config.
Environment and steps to reproduce

Set-up:
on my local Macbook using ct to transpile, then gcloud commands to create in GCP.
YAML config (input):

storage:
  files:
    - path: /opt/cruise-control/config/cruisecontrol.properties
      filesystem: root
      mode: 420
      contents:
        inline:  This represents file content.
    - path: /usr/server.key
      filesystem: root
      mode: 420
      contents:
        inline:  This represents additional content for a second file.
systemd:
  units:
    - name: docker.service
      enabled: true

(etc ... more follows that is not likely relevant, as the only difference between the working and non-working is that second "server.key" file to be included)
ct --platform=gce < yaml_config > config.ign
gcloud compute instances create \"$FLATCAR\" --project \"$PROJECT_NAME\" --zone=$ZONE --image-project kinvolk-public --image-family flatcar-stable --boot-disk-device-name=\"${FLATCAR}-boot-disk\" --machine-type=$INSTANCE_TYPE --tags=$FLATCAR --boot-disk-type=pd-standard $NETWORK_TAGS --metadata-from-file user-data=$TRANSPILED_CONFIG_FILE

Task:
Building a flatcar VM in GCP, and include two files to be written to disk (i.e. in the example above a cruisecontrol.properties file and a server.key file).

Action(s):
Transpile works, as does building the VM in GCP.  Flatcar runs fine IF I only have a single file included.  Adding a second one (another inline, or remote) breaks the flatcar VM and it just keeps rebooting.

Error:
VM console log section that is the first deviation/failure message that I can see when compared to the working version.

[    5.802454] systemd[1]: Finished /sysroot/boot.
[    5.806341] i[    5.986273] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
gnition[556]: fi[    5.988646] ext4 filesystem being mounted at /mnt/oem366638868 supports timestamps until 2038 (0x7fffffff)
les: createFilesystemsFiles: createFiles: op(7): [started]  writing file "/etc/hosts"
[    5.817302] ignition[556]: files: createFilesystemsFiles: createFiles: op(7): oem config not found in "/usr/share/oem", looking on oem partition
[    5.822237] ignition[556]: files: createFilesystemsFiles: createFiles: op(7): op(8): [started]  mounting "/dev/disk/by-label/OEM" at "/mnt/oem702706706"
[    5.827251] systemd[1]: mnt-oem366638868.mount: Succeeded.
[    5.831235] ignition[556]: files: createFilesystemsFiles: createFiles: op(7): op(8): [finished] mounting "/dev/disk/by-label/OEM" at "/mnt/oem702706706"
[[0;1;31mFAILED[0m] Failed to start [0;1;39mIgnition (files)[0m.
[    5.836239] ignition[556]: files failedFull config:
[    5.836440] ignition[556]: {
See 'systemctl status ignition-files.service' for details.
[    5.842244] ignition[556]:   "ignition": {
[[0;1;38;5;185mDEPEND[0m] Dependency failed for [0;1;39mPopu…orcx store to satisfy profile[0m.
[    5.846198] ignition[556]:   [    6.030579] audit: type=1130 audit(1626783998.866:28): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=ignition-files comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=failed'
  "config": {},
[[0;1;38;5;185mDEPEND[0m] Dependency failed for [0;1;39mInitrd Default Target[0m.
[[0;32m  OK  [0m] Stopped [0;1;39mDispatch Password …ts to Console Directory Watch[0m.

Expected behavior
not boot looping =)
Additional information
Weird how I can use the exact same YAML file and build/deploy script, the only difference is when I add that second file path.  Does not matter if the source is inline or remote, a full file or a simple line of text -- the result is that the machine cannot boot.

You can use multiple files entries, what seems to trigger the bug here is your config attempting to write into /usr. /usr is read-only and can't be modified on a running system.
I'll have to look deeper to figure out why its bootlooping. Maybe we should catch attempts to store files to /usr and fail at transpilation time.

Thanks!  That makes a lot of sense.  I was sure it was a bug!  =)
That would be fantastic if you could spew an error instead of rebooting.


Ruby program for Hello world

Hello world program in Ruby language.

Thanks! Make sure to give the repository a star and share about it as well! You can tweet about the repository as well using the tweet badge on the Readme file or website!

Starred the repo! :)


ConvertTo-PnPPage works on test tenant but does not work on production environment.

ConvertTo-PnPPage powershell command works perfectly on test tenant but does not work on prod environment. I have tried different versions of pnp module i.e 1.10.x, 1.10.x nightly, 1.3.0, 1.4.0
Every time it throws below error
ConvertTo-PnPPage : Could not find file 'PnP.Framework.resources'.
At C:\Dentsu\NEON\Migrate_Pages_Prod.ps1:21 char:20

...  $newPage = ConvertTo-PnPPage -PublishingPage -Identity $page.FieldVa ...

            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

CategoryInfo          : WriteError: (:) [ConvertTo-PnPPage], FileNotFoundException
FullyQualifiedErrorId : EXCEPTION,PnP.PowerShell.Commands.Pages.ConvertToClientSidePage

Can you try using PowerShell 7 and check and let us know ?

Closing due to no response received


Output for tfrecords files is not reproducible

Environment info
Operating System:
OS: Debian testing stretch
Kernel: x86_64 Linux 4.5.0-1-amd64
Tensorflow
Version: 0.9.0
Steps to reproduce
Running this script gives a ~/test.tfrecords file.
Running the script repeatedly gives different hashes for the file.
import os
import argparse

import tensorflow as tf
import numpy as np

MAXIMUM_SEQUENCE_LENGTH = 1000

def export(data, name, args):
    num_examples = len(data)

    fh_output = os.path.join(os.path.expandvars(args.output_directory), name + '.tfrecords')
    print('Writing', fh_output)
    writer = tf.python_io.TFRecordWriter(fh_output)
    for index in range(num_examples):
        ## prepare things to save
        seq_x1 = data[index][0].tostring()
        seq_x2 = data[index][1].tostring()

        example = tf.train.Example(features=tf.train.Features(feature={
            'x1': _bytes_feature(seq_x1),
            'x2': _bytes_feature(seq_x2),
            }))
        writer.write(example.SerializeToString())
    writer.close()

def _int64_feature(value):
    return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))

def _bytes_feature(value):
    return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))

if __name__ == '__main__':
    tf.set_random_seed(11)
    np.random.seed(11)

    parser = argparse.ArgumentParser()
    parser.add_argument('--output_directory', type=str, default='$HOME/', help='Directory to write the converted result')
    # parser.add_argument('--validation_size', type=int, default=3000, help='Number of examples to separate from the training data for the validation set.')
    args = parser.parse_args()

    train = [( np.zeros(1000), np.ones(1000), 500 )]

    ## Write data to training file
    export(train, 'test', args)

I wonder why I cannot produce the same binary file again, if I rerun the script. Does this mean the content is different? I didn't find any option to set a seed either.

When I run this program I get the same result every time, with identical md5sum hashes.  What happens if you print the protobuf, instead of using SerializeToString(), are they different?

I am very sorry, in my haste, I seem to have given you a bad toy example. Very strangely, in my setup if I execute a slightly modified example from above, I get different hashes:
example = tf.train.Example(features=tf.train.Features(feature={ 'x1': _bytes_feature(seq_x1), 'x2': _bytes_feature(seq_x2), 'x3': _bytes_feature(seq_x2), }))
It seems that the protobuf is identical though, I have attached an example of the printed output of example (with x1,x2,x3), for which I obtain identical hashes for the protobuf in all runs.
print(example) > test.txt
test.txt
md5sum test.tfrecords test.txt
1630f4da9a52856ac31e84fd482aebe0  test.tfrecords
9e9cda7528deb2e7aa358b39c56b2f1d  test.txt
md5sum test.tfrecords test.txt
e0af3c621d636f147e91582a00cc0834  test.tfrecords
9e9cda7528deb2e7aa358b39c56b2f1d  test.txt
The hashes for the tfrecords file start to repeat themselves as well. I am not entirely sure, whether a similar pattern happens for the x1, x2 case, with lower frequency, though.
Can this be a problem in the following line then?
writer.write(example.SerializeToString())
Can you reproduce the issue with the updated example?
I could also try with a more up-to-date version of tensorflow alternatively.

Sorry for the latency, just got a chance to run your updated program, with the three examples.  I ran it 6 times and got identical output files each time.
You say you always see the same values printed by
print(example)
Have you verified that by taking a checksum of that printed output?
If that is true, yet the example.SerializeToString() output varies, I would suspect a problem not with TensorFlow, but with the python implementation of protocol buffers that you're using, or maybe the filesystem.  It would be good to track down exactly which bytes are varying, and whether they're semantically significant (i.e. do they effect the reparsed value).

Ok. I managed to reproduce the issue on a digital ocean droplet. I have attached all the files, hope that helps to clarify things.
System info:
4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016
Ubuntu 16.04 xenia
Steps to reproduce:
see command_info and issue.txt
I have also included the tfrecord files in the zip folder.
command_info.txt
issue.txt
issue.zip

Doh!  Sorry for not seeing the issue here sooner.  tf.train.example is a protocol message whose internal representation uses some kind of map.  The order in which the map entries are stored in serializied-format is non-deterministic, so that's where your checksum differences come from, even though the ASCII serializations are identical.  This should be clear enough if you squint at your example1.tfrecords and example2.tfrecords: You can see three records inside which look approximately the same, but are in different order.
The non-deterministic ordering of maps inside protobufs is a feature of the protobuf implementation, not TensorFlow.

Did you find a solution for reproducibility of TFRecords creation, especially when it is multithreaded ?
Thanks !


Styling the redirect page

Hey,
Is there a way to style the redirect page? I know there is a small flash with a 0 second delay for those who have js enabled, but for those who do not could I specify a layout to use?
Thanks,
Dave

This would be great.  If instead of hard-coding the redirect page into lib/jekyll-redirect-from/redirect_page.rb the user could specify a _layout to use it'd allow much more easy and flexible customization.

+1


Add Simeons Office

Simeons Office is missing teleports!

I actually mean Solomons office m(


Show Aggregated Perf Results in Perf Graph View

Problem Description
With https://github.com/AdoptOpenJDK/openjdk-test-tools/issues/73, we've added the ability to aggregate the perf results from multiple iterations. We should update Perf Graph View to use and display aggregated perf results in order to comply with the new design.
Proposed Changes
Besides the info that's already displayed, show all aggregated data for each build:

max
min
median
stddev
CI
iteration

Assigned Contributors
I'll be working with Sophia (@sophiaxu0424) from my team to add this feature.

[ ]  make sure one Perf graph (i.e., ODM, SPECjbb, DT3, etc) works with the updated data structure

I have updated ODM.jsx and added extra info from aggregatiInfo set,
having a new graph looks as below:

@piyush286 @llxia

Closing this issue since all related work to this is completed.

Only ODM and AcmeAir graphs are updated to aggregated perf results. Liberty and SPECjbb2015 graphs haven't been updated. Since we aren't really using any perf graphs for perf monitoring due to their current limitations, we can probably do fix those after working on #118.


[sw/opentitantool] Add bitstream ROM splicing capabilities to load_bitstream command

Problem
The opentitantool load_bitstream command loads the FPGA bitstream into the FPGA. By default the FPGA bitstreams contain the test ROM. Unfortunately, in order to run mask ROM E2E tests, the bitstream must contain the mask ROM, not the test ROM. Fortunately, a utility script exists to splice either the test or mask ROM into the bitstream, before it is loaded to the FPGA. However, opentitantool is not aware of this tooling, and therefore the splicing script must be run manually, prior to invoking opentitantool to load the bitstream.
Solution
To make address this issue, the opentitantool load_bitstream command can be expanded to include options for splicing the correct ROM image into the bitstream, before it is loaded on the FPGA.

I'm not completely against this, but don't really understand it. Naively, I'd have thought that "produce a bitstream with the stuff you want" is part of the build step and "now load that bitstream onto the FPGA" is part of the run step. This seems like we're slightly conflating the two?

Ultimately, we want to use opentitantool's load_bitstream command to flash FPGA bitstreams. Currently, you can provide this command with a file path to the bitstream you want to load (it is just minorly inconvenient to run the splicing script manually and then load, I actually have aliases to do so that I would like to replace). So there really are two options here:

embed the splicing functionality as a sub-command to the load_bitstream command to splice the most up to date test/mask ROM images into an existing bitstream (whose path you provide to the command), or

bring bitstream building under bazel control (like Verilator, by having bazel just kick off the right fusesoc command), and have bazel kick off the splicing to produce two bitstreams (one each with test and mask ROM images), then just provide the correct bitstream to the opentitantool load_bitstream command.

Either are ok with me, and I definitely see your point regarding separation of build and load phases. In either case, I think it is important we provide a mechanism to kick off the splicing (through bazel or opentitantool, to keep things centralized) without forcing an entire bitstream re-build, so we can enable fast ROM SW iteration. That is, providing a mechanism to just build and splice ROM images into old bitstreams.

Right, that makes sense. For my 2 cents, I'd probably choose option (2). And it's probably actually easier to implement :-)

I think this is better handled by the build system/bazel. Closing this in favor of #13804. Please reopen if you think otherwise. Thanks!


Update description of pig-latin algorithm

Added a sentence to clarify one of the test requirements of the pig-latin challenge in the intermediate algorithm scripting folder of the Javascript Algorithms And Data Structures Certification.

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX

@Kenford20 thanx for opening this pr and welcome to FCC.
I'm not sure the rule you have added is actually a rule. Can you provide a source?

@Kenford20, everything LGTM! Thank you for helping us improve this section of the curriculum. It's been some time since I completed this challenge, so I had to do some digging around the forums. Seems like a test for this was added in beta but the description was never updated. Hopefully with your addition users won't have as much trouble with this challenge. Looking forward to your next contribution! :+1: :+1:

@nikrb, sorry! I saw that you assigned yourself and I didn't mean to merge without your feedback. Was on autopilot there after leaving a comment. Hope you don't mind. Will be more careful the the future!

@nikrb Yes, it is not actually a rule. It was just meant to clarify the last test in the challenge that states "should handle words with no vowels." As @scissorsneedfoodtoo has mentioned, there was confusion with other campers as well in the forums. After revisiting this challenge, it seems the test has been modified, and the update to the description is no longer necessary (I think? since I was able to take out the portion of my solution that handled that last test case). But I guess it can't hurt :) Thanks guys!

@scissorsneedfoodtoo no worries dude, it's all good 😄


PUT/GET/DELETE lifecycle configurations

https://github.com/opensds/multi-cloud/issues/694
change:
1.PUT/GET/DELETE lifecycle rules, API should be compatible with AWS S3. Synchronously modify the sending parameters
2.Registered backend support YIG

@wisererik @anvithks please review and merge.

@jimccfun Please describe what this PR is doing in the description. It is difficult to review without any context.

@jimccfun Please create an issue in the Dasboard repository for this PR so that we can track the overall changes.

@anvithks  This PR is associated with the issue of multi-cloud, and I don't think it is necessary to create another issue in dashboard

@jimccfun We are creating issues in all the repositories to track progress of work. For any future requirements please create issues. Even if there are issues created in the multi-cloud repo we can link them to these issues and refer them in PRs.

@jimccfun IPlease add more screenshots of the pages where the changes are made and how the screens look like after adding a YIG Backend.

@anvithks In this PR I have linked the issue that was created in multi-cloud, so do I need to create another issue in dashboard?

@anvithks In this PR I have linked the issue that was created in multi-cloud, so do I need to create another issue in dashboard?

Yes we are tracking issues in the repository as well.

@nguptaopensds Please review

@kumarashit @wisererik please review and approve


dhcp Chef-13

Deprecated features used!
 method access to node attributes (node.foo.bar) is deprecated and will be removed in Chef 13, please use bracket syntax (node["foo"]["bar"]) at 2 locations:
   - /tmp/kitchen/cache/cookbooks/dhcp/attributes/default.rb:39:in `from_file'
   - /tmp/kitchen/cache/cookbooks/dhcp/attributes/default.rb:49:in `from_file'
  See https://docs.chef.io/deprecations_attributes.html for further details.
 Cloning resource attributes for directory[/etc/dhcp/shared_networks.d] from prior resource
Previous directory[/etc/dhcp/shared_networks.d]: /tmp/kitchen/cache/cookbooks/dhcp/recipes/_config.rb:55:in `block in from_file'
Current  directory[/etc/dhcp/shared_networks.d]: /tmp/kitchen/cache/cookbooks/dhcp/providers/shared_network.rb:26:in `block in class_from_file' at 1 location:
   - /tmp/kitchen/cache/cookbooks/dhcp/providers/shared_network.rb:26:in `block in class_from_file'
  See https://docs.chef.io/deprecations_resource_cloning.html for further details.
 Cloning resource attributes for directory[/etc/dhcp/shared_networks.d] from prior resource
Previous directory[/etc/dhcp/shared_networks.d]: /tmp/kitchen/cache/cookbooks/dhcp/providers/shared_network.rb:26:in `block in class_from_file'
Current  directory[/etc/dhcp/shared_networks.d]: /tmp/kitchen/cache/cookbooks/dhcp/providers/shared_network.rb:26:in `block in class_from_file' at 1 location:
   - /tmp/kitchen/cache/cookbooks/dhcp/providers/shared_network.rb:26:in `block in class_from_file'
  See https://docs.chef.io/deprecations_resource_cloning.html for further details.
 Cloning resource attributes for template[/etc/dhcp/shared_networks.d/list.conf] from prior resource
Previous template[/etc/dhcp/shared_networks.d/list.conf]: /tmp/kitchen/cache/cookbooks/dhcp/libraries/helpers.rb:45:in `write_include'
Current  template[/etc/dhcp/shared_networks.d/list.conf]: /tmp/kitchen/cache/cookbooks/dhcp/libraries/helpers.rb:45:in `write_include' at 1 location:
   - /tmp/kitchen/cache/cookbooks/dhcp/libraries/helpers.rb:45:in `write_include'
  See https://docs.chef.io/deprecations_resource_cloning.html for further details.
 Cloning resource attributes for directory[/etc/dhcp/classes.d] from prior resource
Previous directory[/etc/dhcp/classes.d]: /tmp/kitchen/cache/cookbooks/dhcp/recipes/_config.rb:55:in `block in from_file'
Current  directory[/etc/dhcp/classes.d]: /tmp/kitchen/cache/cookbooks/dhcp/providers/class.rb:6:in `block in class_from_file' at 1 location:
   - /tmp/kitchen/cache/cookbooks/dhcp/providers/class.rb:6:in `block in class_from_file'
  See https://docs.chef.io/deprecations_resource_cloning.html for further details.
 Cloning resource attributes for directory[/etc/dhcp/classes.d] from prior resource
Previous directory[/etc/dhcp/classes.d]: /tmp/kitchen/cache/cookbooks/dhcp/providers/class.rb:6:in `block in class_from_file'
Current  directory[/etc/dhcp/classes.d]: /tmp/kitchen/cache/cookbooks/dhcp/providers/class.rb:6:in `block in class_from_file' at 1 location:
   - /tmp/kitchen/cache/cookbooks/dhcp/providers/class.rb:6:in `block in class_from_file'
  See https://docs.chef.io/deprecations_resource_cloning.html for further details.
 Cloning resource attributes for template[/etc/dhcp/classes.d/list.conf] from prior resource
Previous template[/etc/dhcp/classes.d/list.conf]: /tmp/kitchen/cache/cookbooks/dhcp/libraries/helpers.rb:45:in `write_include'
Current  template[/etc/dhcp/classes.d/list.conf]: /tmp/kitchen/cache/cookbooks/dhcp/libraries/helpers.rb:45:in `write_include' at 1 location:
   - /tmp/kitchen/cache/cookbooks/dhcp/libraries/helpers.rb:45:in `write_include'
  See https://docs.chef.io/deprecations_resource_cloning.html for further details.
 Cloning resource attributes for directory[/etc/dhcp/subnets.d/] from prior resource
Previous directory[/etc/dhcp/subnets.d/]: /tmp/kitchen/cache/cookbooks/dhcp/providers/subnet.rb:8:in `block in class_from_file'
Current  directory[/etc/dhcp/subnets.d/]: /tmp/kitchen/cache/cookbooks/dhcp/providers/subnet.rb:8:in `block in class_from_file' at 1 location:
   - /tmp/kitchen/cache/cookbooks/dhcp/providers/subnet.rb:8:in `block in class_from_file'
  See https://docs.chef.io/deprecations_resource_cloning.html for further details.
 Cloning resource attributes for template[/etc/dhcp/subnets.d/list.conf] from prior resource
Previous template[/etc/dhcp/subnets.d/list.conf]: /tmp/kitchen/cache/cookbooks/dhcp/libraries/helpers.rb:45:in `write_include'
Current  template[/etc/dhcp/subnets.d/list.conf]: /tmp/kitchen/cache/cookbooks/dhcp/libraries/helpers.rb:45:in `write_include' at 1 location:
   - /tmp/kitchen/cache/cookbooks/dhcp/libraries/helpers.rb:45:in `write_include'

@jmccann i almost have chef 13 compat done
https://github.com/sous-chefs/dhcp/pull/81 finishes all the of deprecation warnings.
One thing i have not gotten is the LWRPs must not notify resources outside of them which means the write_include method in the helpers needs to be reworked.  Not sure the best way to do that

@shortdudey123 it looks like everything is written Chef-10-style.
https://github.com/sous-chefs/dhcp/blob/master/providers/subnet.rb#L12-L33
you should turn on use_inline_resources in those providers.  then just drop all the updated_by_last_action junk entirely.
see:  https://gist.github.com/lamont-granquist/8cda474d6a31fadd3bb3b47a66b0ae78
if these providers are using non-use-inline-resources in order to send notifications to external resources in recipe context (bubble up) then as of chef 12.9.x that works in poise-like ways:
https://gist.github.com/lamont-granquist/b569d0086d19739ebeb3e8c7ab1d65bc
if you need to drop support for chef versions older than 12.9.x i would highly encourage that rather than continuing to write broken resources (and all the code here is broken and buggy it needs to either construct resources directly with Chef::Resource::Whatever.new or use build_resource or use action :nothing, then needs to force running the resource at compile_time and then needs to pull the updated_by_last_action state off of the sub-resource in order to apply it to the wrapping resource correctly -- pretty certain notifications off of these LWRPs don't work correctly and aren't tested).

Thats what i figured i would have to do.  I just just trying to avoid ripping it apart which doesn't look like it will work out the way i thought.
Bubbling up the notifications like the gist you posted seems pretty interesting would appears like thats the right way to go


Stop wrapping keyboard_types::Modifiers.

This was being wrapped to add a small number of extension methods which can be added via a trait instead.
The resulting code is smaller and makes it clear what is different from the upstream keyboard_types::Modifiers.

I think there's a reasonable argument to be made that we can get rid of ModifiersExt as mod.shift() should really be mod.is_shift() and could just as easily be mod.contains(Modifiers::SHIFT) and we don't really lose much, if anything.

In https://github.com/pyfisch/keyboard-types/issues/7, it looks like a PR to add these methods upstream would be accepted, but doesn't exist yet.
Although the author appears to be no longer active, unfortunately. I suspect keyboard-types might need to move under servo?

I've got an email drafted to one of the winit folks suggesting it move to rust-windowing


Mac OS system color pallete

Hey there!
I create a open-color Mac OS pallete using ColorTools,

It was automatically generated using the .ase file provided.
This may be useful for someone. 😃
Download open-color_1.4.0.clr

Thanks! I added it to the download menu.


Upgrade to airflow 2.0

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.jiangxin seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


bug/.docx figures are ignored by partition

Describe the bug
The text inside shapes is not extracted using unstructured.partition. The shapes are ignored altogether.
To Reproduce
In Microsoft Word, create a shape and enter some text into it.

Expected behavior
The text inside the shape shoud be returned by
from unstructured.partition.auto import partition
output = partition("path-to-my-file.docx")
print(output)
# something like  >>>{"Title": "here"}

Environment Info
N/A
Additional context
For example Word2md extracts text in shapes per default
https://github.com/benbalter/word-to-markdown

We've merged a feature which detects text inside inline shapes (shapes anchored as character) and includes them into paragraph's text.

e.g. in such document you'd get two elements with the following text respectively:

"Paragraph with single <inline-image> within."
"Paragraph with <inline-image1> and <inline-image2> within."

This PR doesn't address floating shapes, for now they're not supported. Not detecting text inside floating shapes should be considered a non-existing feature rather then a bug and could be discussed in under an enhacement issue.

We consider this addition a new feature rather then a bug fix, I've updated the title and tags to reflect this.


No instance for Ordering

Ran into an error while trying to generic derive a record with an Ordering property from Prelude.

Yes, we should add an instance for this.

I can send a PR but I'm not sure the correct way to go about it.  Would adding a SigOrdering and SOrdering constructor be correct or to use the SProd/SigProd similar to here https://github.com/purescript/purescript-generics/pull/28/files ?

No, I think we'd just derive an instance here.

@bsermons yeah, deriving will work - I asked you to open the issue here as defining it anywhere else would result in an orphaned instance. :)
Well, aside from the Prelude, but the dependencies don't work out that way.

I'm happy to make the change, by the way, I'm going through the core libraries again and updating for state of the art 0.9 psc.

Ok great, just wanted to offer help in case you guys were busy on other things.  Thanks for everything.

This went into the current v1 release candidate :+1:


need to upgrade node-gyp

distutils has been removed in Python v3.12. we need to upgrade version of node-gyp (to v10+)
› Traceback (most recent call last):
  File<EMAIL_ADDRESS>line 42, in <module>
  import gyp  # noqa: E402
  ^^^^^^^^^^
  File<EMAIL_ADDRESS>line 9, in <mod…
  import gyp.input
  File<EMAIL_ADDRESS>line 19, in <modul…
  from distutils.version import StrictVersion
  ModuleNotFoundError: No module named 'distutils'
  Error: `gyp` failed with exit code: 1
      at ChildProcess.onCpExit (xxxx\node_modules\@electron\rebuild\n…

An unhandled rejection has occurred inside Forge:
Error: node-gyp failed to rebuild 'xxxxx2\node_modules\robotjs'
at ChildProcess.<anonymous> (xxxxx\node_modules\@electron\rebuild\lib\module-type\node-gyp\node-gyp.js:118:24)
    at ChildProcess.emit (node:events:537:28)
    at ChildProcess._handle.onexit (node:internal/child_process:291:12)

can confirm same issue, a temp fix is to include the latest node-gpy as a devDependency in YOUR package.json
npm install node-gyp --save-dev should install 10.0.1

Hi,
I'm struggling to get electron-rebuild to work due to node-gyp version and Python version (as others reported above).
I've tried the so-called temp fix by adding the devDependency in my package.json to the latest node-gyp but it's not getting used by electron-rebuild.
Is there any other known workaround for this issue?

As references in https://github.com/nodejs/node-gyp/issues/2869#issuecomment-1966234393, node-gyp v10+ solves this error; distutils was removed from python 3.12 as mentioned here, which causes this error.
A way to fix this is to add an overrides to your package.json.
In my case (using pnpm) on the highest level in my package.json:
  "pnpm": {
    "overrides": {
      "node-gyp": "^10.0.0"
    }
  },

Do this according to how your package manager handles overrides.
If your package manager does not update correctly, check your lock file and confirm @electron/rebuild has the depency node-gyp set to some 10+ version, at the time of writing 10.0.1. If this is the case, try deleting your lockfile and have it re-generate with e.g. pnpm i or equivalent in your package manager.

For reference, with pnpm my relevant lockfile info looks like this:

  /<EMAIL_ADDRESS>    resolution: {integrity: sha512-zF4x3QupRU3uNGaP5X1wjpmcjfw1H87kyqZ00Tc3HvriV+4gmOGuvQjGNkrJuXdsApssdNyVwLsy+TaeTGGcVw==}
    engines: {node: '>=12.13.0'}
    hasBin: true
    dependencies:
      '@malept/cross-spawn-promise': 2.0.0
      chalk: 4.1.2
      debug: 4.3.4
      detect-libc: 2.0.2
      fs-extra: 10.1.0
      got: 11.8.6
      node-abi: 3.56.0
      node-api-version: 0.2.0
      node-gyp: 10.0.1
      ora: 5.4.1
      read-binary-file-arch: 1.0.6
      semver: 7.6.0
      tar: 6.2.0
      yargs: 17.7.2
    transitivePeerDependencies:
      - supports-color
    dev: true

overrides didn't work for me, but this did:
pip install setuptools

or possibly
python3 -m pip install --break-system-packages setuptools

can confirm same issue, a temp fix is to include the latest node-gyp as a devDependency in YOUR package.json npm install node-gyp --save-dev should install 10.0.1

thx for temp fix actually

pip install setuptools

Thanks, this solutions works for me on windows

overrides didn't work for me, but this did:
pip install setuptools

or possibly
python3 -m pip install --break-system-packages setuptools

This works for me. Thanks @verhovsky

What's the status on this?
It looks like #1128 and #1152 are both proposed fixes.


Ho to deal with this Message : TFLite interpreter needs to link Flex delegate in order to run the model since it contains the following Select TFop(s): Flex ops: FlexTensorListReserve, FlexTensorListSetItem, FlexTensorListStack

1. System information

Linux Ubuntu 16.04
TensorFlow installation 2.17

I want to convert this model and check if it could run on a Coral Edge TPU
2. Code
num_classes = 2
base_model = tf.keras.applications.MobileNetV2(
    include_top=False, weights='imagenet', input_tensor=None,
    input_shape=input_shape,
    pooling=None,
)
for layer in base_model.layers:
    layer.trainable = False
base_model.summary()
cnn = models.Sequential()
cnn.add(base_model)
cnn.add(layers.GlobalAveragePooling2D())
cnn.add(layers.Dropout(0.2))
base_model.trainable = False
model = models.Sequential()
print(full_input_shape)
model.add(layers.TimeDistributed(cnn, input_shape=full_input_shape))
model.add(layers.LSTM(nbr_frame, return_sequences=True))
model.add(layers.TimeDistributed(layers.Dense(nbr_frame, activation='relu')))
model.add(layers.Flatten())
model.add(layers.Dense(164, activation='relu', name="filter1"))
model.add(layers.Dropout(0.2))
model.add(layers.Dense(24, activation='sigmoid', name="filter2"))
model.add(layers.Dropout(0.1))
model.add(layers.Dense(num_classes, activation="sigmoid", name="last"))
rms = optimizers.RMSprop()
metrics = [tf.keras.metrics.CategoricalAccuracy('accuracy', dtype=tf.float32)]
loss = tf.keras.losses.CategoricalCrossentropy()
model.compile(
    loss=loss,
    optimizer= rms,
    metrics=metrics
)
#####.....training the model ... ######
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.target_spec.supported_ops = [
  tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.
  tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.
]
converter.experimental_new_converter = True
tflite_model = converter.convert()
open("fights.tflite", "wb").write(tflite_model)

I am getting this message (I have no idea how to deal with this, I didn't find any relevant documentation about this message):
W0000 00:00:1728657905.522324    4234 tf_tfl_flatbuffer_helpers.cc:392] Ignored output_format.
W0000 00:00:1728657905.522345    4234 tf_tfl_flatbuffer_helpers.cc:395] Ignored drop_control_dependency.
2024-10-11 16:45:05.522588: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: /tmp/tmpwe7b0rla
2024-10-11 16:45:05.543656: I tensorflow/cc/saved_model/reader.cc:52] Reading meta graph with tags { serve }
2024-10-11 16:45:05.543697: I tensorflow/cc/saved_model/reader.cc:147] Reading SavedModel debug info (if present) from: /tmp/tmpwe7b0rla
2024-10-11 16:45:05.843740: I tensorflow/cc/saved_model/loader.cc:236] Restoring SavedModel bundle.
2024-10-11 16:45:06.979508: I tensorflow/cc/saved_model/loader.cc:220] Running initialization op on SavedModel bundle at path: /tmp/tmpwe7b0rla
2024-10-11 16:45:07.522614: I tensorflow/cc/saved_model/loader.cc:462] SavedModel load for tags { serve }; Status: success: OK. Took 2000031 microseconds.
2024-10-11 16:45:10.755926: W tensorflow/compiler/mlir/lite/flatbuffer_export.cc:3463] TFLite interpreter needs to link Flex delegate in order to run the model since it contains the following Select TFop(s):
Flex ops: FlexTensorListReserve, FlexTensorListSetItem, FlexTensorListStack
Details:
	tf.TensorListReserve(tensor<2xi32>, tensor<i32>) -> (tensor<!tf_type.variant<tensor<?x10xf32>>>) : {device = ""}
	tf.TensorListSetItem(tensor<!tf_type.variant<tensor<?x10xf32>>>, tensor<i32>, tensor<?x10xf32>) -> (tensor<!tf_type.variant<tensor<?x10xf32>>>) : {device = "", resize_if_index_out_of_bounds = false}
	tf.TensorListStack(tensor<!tf_type.variant<tensor<?x10xf32>>>, tensor<2xi32>) -> (tensor<10x?x10xf32>) : {device = "", num_elements = 10 : i64}
See instructions: https://www.tensorflow.org/lite/guide/ops_select

Hi, @nassimus26
I apologize for the delayed response, if possible could you please help us with Google colab to reproduce the same bahavior from our end to investigate this issue further from our end ?
Thank you for your cooperation and patience.

tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.

Try removing that line?
In general that is needed for running on CPU, if your model contains TF ops that can't be converted into TF Lite ops.
But I don't think such ops will run on EdgeTPU.  If you remove that line, either it will now just work, or if not you should get some information from the converter about which ops can't be converted.

Hi, @nassimus26
As far I know that  message indicates that your model contains TensorFlow operations that are not natively supported in TFLite and requires the Flex delegate. Try using a simpler architecture that is known to work with Edge TPU. Use the Edge TPU compatibility checker tool and quantize your model to INT8, you'll have to replace TimeDistributed and LSTM with supported operations mentioned here
You can use Flex Delegate to avoid this message TFLite interpreter needs to link Flex delegate in order to run the model since it contains the following Select TFop(s):Flex ops: FlexTensorListReserve, FlexTensorListSetItem, FlexTensorListStack but as you mentioned in the issue temple after converting the model to TFLite you want to run on Coral Edge TPU so which is not recommended for Edge TPU because the Edge TPU does not currently support the Flex Delegate
Please refer this supported operations on the Edge TPU in that TimeDistributed is not supported operations and unidirectional LSTM only supported
If I have missed something here please let me know.
Thank you for your cooperation and patience.


Typo in one of the commands on the page - Get-AzRecoveryServicesBackupJobDetails

Hi,
I believe there is a minor typo on this page. It states one of the commands is Get-AzRecoveryServicesBackupJobDetails , but should be Get-AzRecoveryServicesBackupJobDetail
As the link go to the Get-AzRecoveryServicesBackupJobDetail documentation page.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e50dfe50-c177-f976-753b-9fa689d20f92
Version Independent ID: 7f787259-8ce5-9ba6-8df7-58ab0b89dd5c
Content: Back up and recover Azure VMs with PowerShell - Azure Backup
Content Source: articles/backup/backup-azure-vms-automation.md
Service: backup
GitHub Login: @v-amallick
Microsoft Alias: v-amallick

@darylnz Great observation! I have verified and updated the document as required. The change should reflect soon i.e., in less than 24 hours. Thanks again!


Add test to catch need for JSON output schema update

What would you like to be added:
A test validating that the latest schema version (for SBOM JSON output format) and structures that are marshaled into JSON are matching.
Why is this needed:
Needed so code and schema are always aligned.
Additional context:
The lack of checks caused an extra review step on https://github.com/anchore/syft/pull/1026

Closed as this file validates the syft output against the schema:
https://github.com/anchore/syft/blob/main/test/cli/json_schema_test.go


[FLINK-17058] Adding ProcessingTimeoutTrigger of nested triggers.

What is the purpose of the change
Adding new feature of ProcessingTimeoutTrigger as mention in FLINK-17058 jira page.
Verifying this change
Adding UnitTests to the new code.
Does this pull request potentially affect one of the following parts:
Does not affect anything of the mentioned.
Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? not documented, where should i document it?

The CI build failed because the new files are missing license headers.

@aljoscha thanks for helping on first contribution, added the license headers and fixed the CR changes.

@aljoscha Fixed the CR comments, can u check?

 If you're ok with the changes I would go ahead and merge this.

@aljoscha I personally don't like to use null (or at least wrap it with Objects.isNull), but i guess it is okay because the other Triggers use it too.
You can merge this, thanks for help.

@flinkbot run azure

I agree about nulls in general, but in this case the code was just creating new objects internally, which comes with some overhead.

I merged this! 👌


Update README

Some things to include:

What skills we're looking for (StimulusJS, anything else?)
More info about Ruby for Good
How to contribute
Review setup, make sure it works
Anything else you want to add!

I think the README could move some information to a Wiki on GH. I personally like to make the README very short and optimize for getting the local machine setup. Happy to open a PR for this :)


How to encrypt a field from nested array

{
"_id":"5f841438366150038e3c6840",
"createdOn":"2020-10-09T18:53:47.924+05:00",
"recipientType":"Customer",
"recipients":[
{
"_id":"5f8414383661502db23c6841",
<EMAIL_ADDRESS>},
{
"_id":"5f8414383661502db23c6841",
<EMAIL_ADDRESS>}
],
"__v":0
}
I want to encrypt and decrypt receiver email address from the same data above .. Its inside array
This is what my code looks like
const recipientsModel = new mongoose.Schema({
_memberId: {
type: mongoose.Types.ObjectId,
},
receiver: {
type: String,
},
});
const processedTemplateModel = new mongoose.Schema({
recipientType: {
type: String,
required: true,
},
recipients: {
type: [recipientsModel,
],
},
createdOn: {
type: Date,
default: new Date(),
},
});
recipientsModel.plugin(markFieldsAsPII, {
fields: ['receiver'],
key: process.env.MONGOOSE_PII_KEY,
});
processedTemplateModel.plugin(markFieldsAsPII, {
fields: ['recipients.receiver'],
key: process.env.MONGOOSE_PII_KEY,
});
const getProcessedTemplateModel = (tenantId?: String) => {
return new MultiTenancy(tenantId).getTenantModel(
'ProcessedTemplateMuzz',
processedTemplateModel,
'processedtemplates_muzz',
);
};
This is how i am trying to display decrypted emails
const templates = await getProcessedTemplateModel().find();
It is encrypting but not decrypting .. Can anyone help

Hey there!
I'm afraid the logic for identifying encrypted fields, especially on read, didn't take into account nested fields, be they inside arrays or single nested objects.
We unfortunately do not have time to maintain this anymore, but you should know that MongoDB supports client-side encryption natively now (since 4.2, hoping you're on a recent-enough instance; Enterprise is only required for "automatic encryption"), and that depending on your threat models, server-side, at-rest encryption may be a nicer solution (as it preserves all your querying logic, for instance).
Hoping this helps,

thanks @tdd  : i achieved this through
recipientsModel.plugin(markFieldsAsPII, {
fields: ['receiver'],
key: process.env.MONGOOSE_PII_KEY,
});
processedTemplateModel.plugin(markFieldsAsPII, {
fields: [''],
key: process.env.MONGOOSE_PII_KEY,
});
const getProcessedTemplateModel = (tenantId?: String) => {
return new MultiTenancy(tenantId).getTenantModel(
'ProcessedTemplateMuzz',
processedTemplateModel,
'processedtemplates_muzz',
);
};
Hope it helps anyone who is having this issue

I have mentioned the solution above


bye bye world

What the program do ?

In what programming language it is written?

This isn't very descriptive. What does the program actually do?


GA splunk-otel-ios

Which GDI repository do you wish to GA?
splunk-otel-ios https://github.com/signalfx/splunk-otel-ios
Does the repository follow the latest tagged minor release in GDI specification?

[x] Has an appropriate maintainers team.
https://github.com/orgs/signalfx/teams/gdi-ios-maintainers
[x] Permissions
set correctly.
[x] Branch protection
in place.
[x] Dependencies
appropriately locked down.
[x] GitHub Applications
set up per spec.
[x] Follows the configuration requirements,
if appropriate.
[x] Follows the semantic convention requirements,
if appropriate.
[x] Required Files
in place.

[x] CHANGELOG.md
[x] CODE_OF_CONDUCT.md
[x] CONTRIBUTING.md
[x] .github/CODEOWNERS
[x] LICENSE
[x] README.md

[x] Build status badge
[x] Getting started
[x] Troubleshooting
[x] Link to official Splunk docs
[x] License information

[x] SECURITY.md

[x] Releases
done to spec.
[x] Type specific requirements (remove what doesn't apply)

[x] Instrumentation Library

[x] Documents all supported configuration parameters.
[x] Documents how to configure manual instrumentation.
[x] Documents how to configure log correlation.
[x] Documents minimum supported version of each auto-instrumentation framework.
Hosted at https://github.com/splunk/public-o11y-docs

How long has the GDI repository been public?
Two years
Is the repository known to be used today?
yes
Is there a date by which this approval is needed?**
No
Additional context
None

@pellared this repo is ready for review

@dvernon-splunk, I have removed all checkboxes from the description + updated links to 1.6.0. I will go through verification shortly.

Has an appropriate maintainers team. Setting adjusted:
@signalfx/gdi-instrumentation-admins - set as admins
@signalfx/gdi-ios-approvers - set as write

Fixed by: https://github.com/signalfx/splunk-otel-ios/pull/174

[x] CHANGELOG.md
[x] CODE_OF_CONDUCT.md
[x] CONTRIBUTING.md
[x] .github/CODEOWNERS
[x] LICENSE
[x] SECURITY.md

[x] GitHub Applications
set up per spec.

It is technically fine, but repository is not using CodeCov.

For

[ ] Dependencies
appropriately locked down.

Dependencies

MUST lock the versions of all build dependencies (e.g. libraries, binaries,
scripts, docker images) or vendor them; EXCEPTION: tools that are
available out-of-the-box on the CI runner

for sure it is no done done for action/checkout in https://github.com/signalfx/splunk-otel-ios/blob/a9e464d31bb72398e9953f2b9bf708f087f52d95/.github/workflows/ci.yml#L12
I am not sure how to verify swift dependencies, could you please show me an example defining exact dependency?

MUST enable Dependabot alerts

MUST grant access to alerts for the approvers and maintainers teams

I have added maintainers team here

MUST enable Dependabot security updates

MUST configure Dependabot version updates

We only have one dependency, Swifter, which is found in the Package.resolved file https://github.com/signalfx/splunk-otel-ios/blob/a9e464d31bb72398e9953f2b9bf708f087f52d95/SplunkRumWorkspace/SplunkRumWorkspace.xcworkspace/xcshareddata/swiftpm/Package.resolved#L6 It is locked to 1.5.0

[ ] Releases
done to spec.

MUST have a signature or a checksum with signature for all release artifacts

SHOULD use Splunk signing key

MUST use signed tags

MUST have a tag protection rule
for the release tags (e.g. v*)

MUST state version of OpenTelemetry repository built against if applicable

I do not see any upstream, should be fine

MUST update all examples in the Splunk OpenTelemetry example
repository
that depends on the repository to use the latest release.

There is a ios application: https://github.com/signalfx/tracing-examples/tree/main/ios/Demo Boutique/Demo Boutique which seems to be not update/instrumented by this reposiroty. Please do it, and extend you RELEASING.md to handle it.

https://github.com/signalfx/tracing-examples/tree/main/ios/Demo Boutique

Do you have a plan to GA also: https://github.com/signalfx/splunk-otel-ios-crashreporting ?
If so, I think that separate GA request is needed.

[ ] Real User Monitoring Library

[ ] Documents all supported configuration parameters. - requires update when you copy issues from https://github.com/signalfx/gdi-specification/issues/255#issuecomment-1786677554
[x] Documents how to configure manual instrumentation.
[x] Documents supported instrumentation.
[x] Documents supported platforms.

Do you have a plan to GA also: https://github.com/signalfx/splunk-otel-ios-crashreporting ? If so, I think that separate GA request is needed.

Eventually, but not at this time


Ensure the presence of bottom padding at all times

At all times, there should be bottom padding on the pages so that the footer has air to breathe.

Fixed in d8a913d1f272517773a2c6c93e1b22d51b8b63da.


set userAgentSuffix for documentDb starters

@ZhijunZhao  @aovechki

@yungez,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot


Return response of initial HEAD request

We need this response to determine our caching strategy, specifically we want to know if the response is immutable.
I'm not completely happy with that caching strategy yet but it works for pypi and wheels at a url should be immutable (otherwise you get a hash mismatch when deploying).

This looks good to me but wouldn't you then also want to add additional headers to the initial request?

yes, i'll likely add that in the future, though i still have to figure out how to integrate this; i'm still looking for something that implements http caching semantics on top of reqwest without being a middleware, https://github.com/06chaynes/http-cache/issues/57). For pypi i get away with just checking if the cache control header says "immutable".

Yeah I would like to have something like that too! Im following your issue. :)
Would you be able to look at the other comments I left?


Path to set/map position

Paths can only address values in set/map keys/values. It would be useful to have a way to address keys/values at positions in the set/map.
This is stable because sets/maps have a well defined ordering.
Strawman: use {}, for example given the map:
value: {
  "foo": 42,
  "bar": {
    "baz": true
  },
  17: false
}

address with
> noms show value{0}
42
> noms show value{1}
"bar": {"baz": true}
> noms show value{1}@key
"bar"
> noms show value{1}["baz"]
true
> noms show value{2}
false
> noms show value{2}@key
17

Ditto sets. @aboodman

New strawman:

Always make [42] index by position.
For map keys that are numbers, require [@42].

I think this is better than the inconsistency between lists.

FINAL STRAWMAN:

Make {42} index by position.
Support {42} syntax on lists for indexing by position.
Flourish: support {-1} for indexing the last position (and {-2} for the 2nd last, etc).

lgtm

I think the syntax should be exactly the same for square brackets and curly brackets for lists.

Agree