[FEATURE] Add conversation support to Feedback

Is your feature request related to a problem? Please describe.
It is not clear how to use Feedback scheme to store multi-turn conversation.  It doesn't look like there is support in the web interface to annotate feedback for multi-turn conversations. A lot of Instruction Tuning data is multi-turn and the Feedback scheme only allows to record feedback to a single response.  Other schemes (i.e. Text2Text) also seem to not be suitable.
Describe the solution you'd like
Native support for multi-turn, allowing to annotate feedback for each turn of the conversation.
Describe alternatives you've considered
Getting feedback on each turn individually and then providing context (full conversation) for each turn.  This is not great, as this results in the need to re-read each conversation N times, where N is the number of turns.  Slowing down the progress substantially.
Alternative is storing the complete conversation as text. And using the external tool, like Gradio to annotate.  This diminishes the value of Argilla, as it requires to add external/not integrated tool, disconnected from search / etc.  Storing the conversation as text, instead of structured data also reduces the ability to filter the conversation in the structured way (i.e. by last response).  Alternative is storing the complete conversation as a .json field.  This is also not great, as nearly all tooling of argilla do not have any support for this.
Additional context
Ideally this should be natively supported in the web UI.  But this may be implemented using a tool/plugin outside the current web/gui.  Using Gradio/Chat as a frontend may be a good option.  But either way, this still requires support at the schema level.

Hi @dchichkov ! Thanks for the suggestion! I definitely want to know more about your use case. Can I ask if the number of turns is always the same or if it changes for each conversation?

Thanks @dchichkov! Looking forward to your comments. I think this is an important field type (or extension to the current text field). I believe I briefly comment this with @alvarobartt when we did the langchain callback. In my experience, the number of turns is variable across records. There several example datasets available on the Hub, but @dchichkov if you are looking at a specific dataset/format it would be awesome if you could give us some pointers

Exactly, this was also something we discussed during the ML weekly. Potentially allowing to have something like a 'TextListField' and 'TextQuestionList'

Hi @nataliaElv & @dvsrepo .  An example could be https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K
The format there (this is a single multi-turn conversation) is:
[   {     "id": "000000033471",     "image": "000000033471.jpg",     "conversations": [       {         "from": "human",         "value": "<image>\nWhat are the colors of the bus in the image?"       },       {         "from": "gpt",         "value": "The bus in the image is white and red."       },       {         "from": "human",         "value": "What feature can be seen on the back of the bus?"       },       {         "from": "gpt",         "value": "The back of the bus features an advertisement."       },       {         "from": "human",         "value": "Is the bus driving down the street or pulled off to the side?"       },       {         "from": "gpt",         "value": "The bus is driving down the street, which is crowded with people and other vehicles."       }     ]   },   ...
I'm not sure that this exact format is the best. I think that simply using Markdown with the URL for images is a lot more natural, removes extra complexity and allows more flexibility on the renderer and the dataset side.

In terms of the number of conversations, turns, images, participants, participant names - it varies for every conversation. To put some rough numbers:

Conversations: 100k
Conversation Turns: 1 ... 50, median 5
Turn Length: 1...500 lines of text,  median 2
Participants: 1 .. 10, median 2
Images: 0 .. 7, median 1

+1 for this feature request. I feel like this will be an increasingly important functionality in annotation interfaces.
To give some more input/examples: DeepMind and Anthropic have created similar interfaces for their internal use and it would be great to have an open source option.

In this paper from DeepMind on p. 51 you can see their interface. (Note that this interface also provides snippets for sources, which is not necessary for Argilla I think)

in this paper from Anthropic on p. 5 you can see their interface.

They've always limited the number of turns to a specific N amount (5~). Another important thing in both these studies is they they have human text input + live model text output (which could come from e.g. the Hugging Face inference API or some other LLM APIs), as well as separate annotation boxes to rate each model output (which probably makes things a bit more complicated).
One open-source implementation for chat-based annotation is Meta's Mephisto/ParlAI's chat interface that directly integrates with MTurk. I haven't tested it yet though. It's based on React and probably requires some JS/React knowledge customize  https://github.com/facebookresearch/Mephisto/tree/main/examples/parlai_chat_task_demo

Also +1 for this feature request.  When you think of a chatbot type response with LLMs, we quickly end up with multi-turn conversations.

@MoritzLaurer @ttamg, thanks for the context. Any feedback and context are always welcome.

@dchichkov @ttamg @MoritzLaurer We're currently working towards a ChatField for Argilla datasets that would look similar to the output of our current chat_to_html helper function. Have any of you tried it? If so, I'd love to learn about your experience with it. If you're happy to be contacted drop me an email at<EMAIL_ADDRESS>


Security Vulnerability Issue (Stored XSS)

Describe the bug
Hi,
i have reported a security vulnerabilities about a stored XSS.
you can see all the details here:
https://huntr.dev/bounties/7e1be91d-3b13-4300-8af2-9bd9665ec335/
i hope i was helpful
Steps to reproduce
See link in description for PoC
Screenshots or additional context
No response

Thanks, already fixed.


Allow configFolder to be set via environment variable

I try to keep all my configuration in XDG standard inside the ~/.config directory. Could you allow setting --configFolder as an environment variable? e.g. GCALCLI_CONFIG_DIR

Should be possible.  I'll take a look

That'll be great thanks

Ideally we split  up the config (the rc file) and cache data (oauth2 and actual cache) and stick them into $XDG_CONFIG_DIR and $XDG_CACHE_DIR respectfully. Those variables are exactly for that purpose so that means $GCALCLI_CONFIG_DIR is no longer needed.

Would support XDG used by default for new installations.
The xdg Python package makes it easy to use the XDG standard.

That sounds reasonable to me.  I'd like to convert the rc file format itself to something standard (instead of just holding options in text, there's an issue for this).

Is this being worked on, or should I create a patch for it?
I took a quick glance at the code, and it should be quite straightforward to fix it. I can take a further look at it on Wednesday, in 2 days, and submit a PR.
I'd also like to address the "issue" of --config complaining about being ambiguous, and the fact that using the --config parameter expects one file name, while the default one requires no .gcalcli_ prefix.

Is there any follow up on this? Would also like to have this functionality and to not clutter up the home dir.

Is there any follow up on this? Would also like to have this functionality and to not clutter up the home dir.

Never mind, found the option --config-folder which does exactly this.


increment version to 2.14

Description
Increment version to 2.14
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

I think I made this obsolete when I merged #309 ... rebasing should make this PR a no-op.

Rebased to 2.14 which auto-closed as all changes were there. :)


Web Accessibility Issues on DummyGram

Description
Summary
I have scanned the DummyGram webpage using axe DevTools and found 90 web accessibility issues that affect the user experience and accessibility of the website. These issues include:

Buttons must have discernible text (53 instances)
Elements must meet minimum color contrast ratio thresholds (35 instances)
Form elements must have labels (1 instance)
Interactive controls must not be nested (1 instance)

Expected Result
The webpage should comply with the Web Content Accessibility Guidelines (WCAG) 2.1 AA standards and have no accessibility issues.
Actual Result
The webpage has 90 accessibility issues that violate the WCAG 2.1 AA standards and impair the user experience and accessibility of the website.
Steps to reproduce

Go to DummyGram
Open axe DevTools and run an analysis
Observe the results in the Overview tab

Screenshots

Desktop (please complete the following information)

Browser: Brave
Operating System: Windows 10

Smartphone (please complete the following information)
No response
Additional information
I am interested in fixing these issues and making the website more accessible. Please let me know if I can work on this issue and submit a pull request.
Record

[X] I agree to follow this project's Code of Conduct
[X] I have checked the existing issues
[X] I am a participant of a program
[X] I want to work on this issue

Optional information
No response

Description
Summary
I have scanned the DummyGram webpage using axe DevTools and found 90 web accessibility issues that affect the user experience and accessibility of the website. These issues include:

Buttons must have discernible text (53 instances)
Elements must meet minimum color contrast ratio thresholds (35 instances)
Form elements must have labels (1 instance)
Interactive controls must not be nested (1 instance)

Expected Result
The webpage should comply with the Web Content Accessibility Guidelines (WCAG) 2.1 AA standards and have no accessibility issues.
Actual Result
The webpage has 90 accessibility issues that violate the WCAG 2.1 AA standards and impair the user experience and accessibility of the website.
Steps to reproduce

Go to DummyGram
Open axe DevTools and run an analysis
Observe the results in the Overview tab

Screenshots

Desktop (please complete the following information)

Browser: Brave
Operating System: Windows 10

Smartphone (please complete the following information)
No response
Additional information
I am interested in fixing these issues and making the website more accessible. Please let me know if I can work on this issue and submit a pull request.
Record

[x]  I agree to follow this project's Code of Conduct
[x]  I have checked the existing issues
[x]  I am a participant of a program
[x]  I want to work on this issue

Optional information
No response

Hi @usernayeem you can go ahead! Lemme know if I can help as well :) Thanks


Cordova Pushwoosh plugin HWID is changed for iOS and Android?

Hope you are doing well. I am using Pushwoosh cordova plugin for Android and iOS apps. I am just wondering if hwid is always unique for iOS device, so I can use to detect special ios device with it? It will be changed for any circumstances? In that case, I will not be able to use hwid as unique device id for iOS device. Thanks for your great help.
Sincerely

Pushwoosh uses the following HWID algorithm:

If you link to AdSupport.framework it will user Advertisement Identifier. This identifier is unique for the device and is the same for all apps installed. The only way for this identifier to change is when user forces it to change in the option deeply buried somewhere in device settings.

If you do not link to AdSupport.framework, Identifier for Vendor would be used. This identifier is unique for the device but will change between the apps if they are signed with different Apple Developer Certificate. Basically it will be different for different Apple Developer Accounts.

You can read more about in the Apple Docs
https://developer.apple.com/library/ios/documentation/AdSupport/Reference/ASIdentifierManager_Ref/index.html#//apple_ref/occ/instp/ASIdentifierManager/advertisingIdentifier:
https://developer.apple.com/library/ios/documentation/UIKit/Reference/UIDevice_Class/index.html#//apple_ref/occ/instp/UIDevice/identifierForVendor
Also some useful links on this identifiers:
http://stackoverflow.com/questions/13855990/are-there-other-factors-besides-device-and-app-vendor-that-determine-the-ident
http://stackoverflow.com/questions/11836225/ios6-udid-what-advantages-does-identifierforvendor-have-over-identifierforadve


[Mono] NuGet version too old on Linux

Hi,
I use omnisharp-roslyn with Emacs on Linux and because Mono bundled with omnisharp doesn't have any other .NET SDKs than 4.5 (and I mostly work with 4.7.2 these days),
I tend to modify the run script to start omnisharp with my locally built Mono from the master branch. Until recently it's been working fine, but with the latest update to MSBuild
16.8, it broke with:
[16:27:28] ERROR: OmniSharp.MSBuild.ProjectLoader, The "ResolvePackageAssets" task failed unexpectedly.
System.MissingMethodException: Method not found: NuGet.ProjectModel.LockFileTarget NuGet.ProjectModel.LockFile.GetTarget(string,string)
  at Microsoft.NET.Build.Tasks.ResolvePackageAssets+CacheReader.CreateReaderFromDisk (Microsoft.NET.Build.Tasks.ResolvePackageAssets task, System.Byte[] settingsHash) [0x0003a] in <1362f3759017415f99456b8adb293671>:0
  at Microsoft.NET.Build.Tasks.ResolvePackageAssets+CacheReader..ctor (Microsoft.NET.Build.Tasks.ResolvePackageAssets task) [0x00015] in <1362f3759017415f99456b8adb293671>:0
  at Microsoft.NET.Build.Tasks.ResolvePackageAssets.ReadItemGroups () [0x00000] in <1362f3759017415f99456b8adb293671>:0
  at Microsoft.NET.Build.Tasks.ResolvePackageAssets.ExecuteCore () [0x00018] in <1362f3759017415f99456b8adb293671>:0
  at Microsoft.NET.Build.Tasks.TaskBase.Execute () [0x0002e] in <1362f3759017415f99456b8adb293671>:0
  at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute () [0x00029] in <f7274894ce154696afba1f0ffcd06434>:0
  at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask (Microsoft.Build.BackEnd.ITaskExecutionHost taskExecutionHost, Microsoft.Build.BackEnd.Logging.TaskLoggingContext taskLoggingContext, Microsoft.Build.BackEnd.TaskHost taskHost, Microsoft.Build.BackEnd.ItemBucket bucket, Microsoft.Build.BackEnd.TaskExecutionMode howToExecuteTask) [0x002fc] in <f7274894ce154696afba1f0ffcd06434>:0

It appears that NuGet 5.2.0 which is used by omnisharp is too old. I fixed it by updating nuget to 5.8.0 (the latest) and rebuilding omnisharp. Everything works perfectly now :)
I tested it only on Linux but I suppose the same might be true on macOS.
Could you please update NuGet to the above-mentioned version? I'm happy to open a PR if needed. Thanks :)

Same issue here.

I seem to be having a similar issue? Mine is a .NET Core 3.1 project:
OmniSharp.MSBuild.ProjectLoader: The "ResolvePackageAssets" task failed unexpectedly.
NuGet.Packaging.Core.PackagingException: Unable to find fallback package folder 'C:\Program Files\dotnet\sdk\NuGetFallbackFolder'.
  at NuGet.Packaging.FallbackPackagePathResolver..ctor (System.String userPackageFolder, System.Collections.Generic.IEnumerable`1[T] fallbackPackageFolders) [0x00064] in <665a1c83623344819944d40680140855>:0
  at Microsoft.NET.Build.Tasks.NuGetPackageResolver..ctor (System.String userPackageFolder, System.Collections.Generic.IEnumerable`1[T] fallbackPackageFolders) [0x00006] in <c1a29c5daf134e4ba6d73cbbae7974d1>:0
  at Microsoft.NET.Build.Tasks.NuGetPackageResolver.CreateResolver (System.Collections.Generic.IEnumerable`1[T] packageFolders) [0x00018] in <c1a29c5daf134e4ba6d73cbbae7974d1>:0
  at Microsoft.NET.Build.Tasks.NuGetPackageResolver.CreateResolver (NuGet.ProjectModel.LockFile lockFile) [0x0002a] in <c1a29c5daf134e4ba6d73cbbae7974d1>:0
  at Microsoft.NET.Build.Tasks.ResolvePackageAssets+CacheWriter..ctor (Microsoft.NET.Build.Tasks.ResolvePackageAssets task) [0x00030] in <c1a29c5daf134e4ba6d73cbbae7974d1>:0
  at Microsoft.NET.Build.Tasks.ResolvePackageAssets+CacheReader.CreateReaderFromDisk (Microsoft.NET.Build.Tasks.ResolvePackageAssets task, System.Byte[] settingsHash) [0x0003a] in <c1a29c5daf134e4ba6d73cbbae7974d1>:0
  at Microsoft.NET.Build.Tasks.ResolvePackageAssets+CacheReader..ctor (Microsoft.NET.Build.Tasks.ResolvePackageAssets task) [0x00015] in <c1a29c5daf134e4ba6d73cbbae7974d1>:0
  at Microsoft.NET.Build.Tasks.ResolvePackageAssets.ReadItemGroups () [0x00000] in <c1a29c5daf134e4ba6d73cbbae7974d1>:0
  at Microsoft.NET.Build.Tasks.ResolvePackageAssets.ExecuteCore () [0x00018] in <c1a29c5daf134e4ba6d73cbbae7974d1>:0
  at Microsoft.NET.Build.Tasks.TaskBase.Execute () [0x0002e] in <c1a29c5daf134e4ba6d73cbbae7974d1>:0
  at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute () [0x00029] in <ea47b2df30ad4051b6681ba57ba83dc9>:0
  at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask (Microsoft.Build.BackEnd.ITaskExecutionHost taskExecutionHost, Microsoft.Build.BackEnd.Logging.TaskLoggingContext taskLoggingContext, Microsoft.Build.BackEnd.TaskHost taskHost, Microsoft.Build.BackEnd.ItemBucket bucket, Microsoft.Build.BackEnd.TaskExecutionMode howToExecuteTask) [0x002fc] in <ea47b2df30ad4051b6681ba57ba83dc9>:0  |
OmniSharp.MSBuild.ProjectLoader: The "ResolvePackageAssets" task failed unexpectedly.
NuGet.Packaging.Core.PackagingException: Unable to find fallback package folder 'C:\Program Files\dotnet\sdk\NuGetFallbackFolder'.
  at NuGet.Packaging.FallbackPackagePathResolver..ctor (System.String userPackageFolder, System.Collections.Generic.IEnumerable`1[T] fallbackPackageFolders) [0x00064] in <665a1c83623344819944d40680140855>:0
  at Microsoft.NET.Build.Tasks.NuGetPackageResolver..ctor (System.String userPackageFolder, System.Collections.Generic.IEnumerable`1[T] fallbackPackageFolders) [0x00006] in <c1a29c5daf134e4ba6d73cbbae7974d1>:0
  at Microsoft.NET.Build.Tasks.NuGetPackageResolver.CreateResolver (System.Collections.Generic.IEnumerable`1[T] packageFolders) [0x00018] in <c1a29c5daf134e4ba6d73cbbae7974d1>:0
  at Microsoft.NET.Build.Tasks.NuGetPackageResolver.CreateResolver (NuGet.ProjectModel.LockFile lockFile) [0x0002a] in <c1a29c5daf134e4ba6d73cbbae7974d1>:0
  at Microsoft.NET.Build.Tasks.ResolvePackageAssets+CacheWriter..ctor (Microsoft.NET.Build.Tasks.ResolvePackageAssets task) [0x00030] in <c1a29c5daf134e4ba6d73cbbae7974d1>:0
  at Microsoft.NET.Build.Tasks.ResolvePackageAssets+CacheReader.CreateReaderFromDisk (Microsoft.NET.Build.Tasks.ResolvePackageAssets task, System.Byte[] settingsHash) [0x0003a] in <c1a29c5daf134e4ba6d73cbbae7974d1>:0
  at Microsoft.NET.Build.Tasks.ResolvePackageAssets+CacheReader..ctor (Microsoft.NET.Build.Tasks.ResolvePackageAssets task) [0x00015] in <c1a29c5daf134e4ba6d73cbbae7974d1>:0
  at Microsoft.NET.Build.Tasks.ResolvePackageAssets.ReadItemGroups () [0x00000] in <c1a29c5daf134e4ba6d73cbbae7974d1>:0
  at Microsoft.NET.Build.Tasks.ResolvePackageAssets.ExecuteCore () [0x00018] in <c1a29c5daf134e4ba6d73cbbae7974d1>:0
  at Microsoft.NET.Build.Tasks.TaskBase.Execute () [0x0002e] in <c1a29c5daf134e4ba6d73cbbae7974d1>:0
  at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute () [0x00029] in <ea47b2df30ad4051b6681ba57ba83dc9>:0
  at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask (Microsoft.Build.BackEnd.ITaskExecutionHost taskExecutionHost, Microsoft.Build.BackEnd.Logging.TaskLoggingContext taskLoggingContext, Microsoft.Build.BackEnd.TaskHost taskHost, Microsoft.Build.BackEnd.ItemBucket bucket, Microsoft.Build.BackEnd.TaskExecutionMode howToExecuteTask) [0x002fc] in <ea47b2df30ad4051b6681ba57ba83dc9>:0  |

Is this related?

@JordanAnthonyKing It doesn't seem to be the same issue. Can you confirm that the folder 'C:\Program Files\dotnet\sdk\NuGetFallbackFolder' exists?

@JoeRobich Hi, I eventually figured this out as a misnomer for an incorrectly configured project.
Thanks


adjust kubectl top nodes for better display with open sidecar

we should show only NAME and the two % columns. right now, we are showing too many columns.

desired:
note the "node / default" shouldn't have "default". this is covered by #4599


Handling of broken Alonzo address missing from the spec

Background
On August 28th, cardano-node 1.29.0 was released. Shortly after, on Sept 5th, this transaction which contains an invalid address was created.
Information about this address:

Its bech32 representation is addr1q9d66zzs27kppmx8qc8h43q7m4hkxp5d39377lvxefvxd8j7eukjsdqc5c97t2zg5guqadepqqx6rc9m7wtnxy6tajjvk4a0kze4ljyuvvrpexg5up2sqxj33363v35gtew
Its binary representation is 015bad085057ac10ecc7060f7ac41edd6f63068d8963ef7d86ca58669e5ecf2d283418a60be5a848a2380eb721000da1e0bbf39733134beca4cb57afb0b35fc89c63061c9914e055001a518c7516
It's a base address address of type <keyhash, keyhash> according to its header
The address fails to be parsed by cardano-address
The complete txbody this tx is contained in is as follows: a40083825820bf4f8f6287993e17f785c949ef287416ba784198bb0ee12b2c7720cbffecd26f0082582052cf971bee4ba1b4e3b32b762e0bc3f7af83700bc8897eddbd77bd425dd28e8300825820bf4f8f6287993e17f785c949ef287416ba784198bb0ee12b2c7720cbffecd26f01018282584e015bad085057ac10ecc7060f7ac41edd6f63068d8963ef7d86ca58669e5ecf2d283418a60be5a848a2380eb721000da1e0bbf39733134beca4cb57afb0b35fc89c63061c9914e055001a518c75161a035377d082583901f4e9e89cc628b9204fd6e2f4ae3e875aa0591fc2eb45b721520d2d22f4e9e89cc628b9204fd6e2f4ae3e875aa0591fc2eb45b721520d2d221a2c651d70021a00031570031a05f5e100 but it may be good to verify this tx body with some different tool to make sure this isn't an chain indexer issue (it could be that Oura and CardanoScan's indexer both happen to have the same bug and mangle the address info in the same way)
If you remove the extra bytes from this address, you end up with the address addr1q9d66zzs27kppmx8qc8h43q7m4hkxp5d39377lvxefvxd8j7eukjsdqc5c97t2zg5guqadepqqx6rc9m7wtnxy6tajjq6r54x9

The next node version (1.30.1) doesn't mention this bug being resolved, so I'm not sure if it sill exists or if it was just left out of the changelogs
What to do?

Update the ledger spec to document how this address should be handled
Update cardano-address to parse this address

@SebastienGllmt Thank you for reporting this issue. We are aware of this problem and a fix is on the way very soon!

Just FYI there appears to be 4 addresses on mainnet affected by this

address
extra bytes

addr1q9d66zzs27kppmx8qc8h43q7m4hkxp5d39377lvxefvxd8j7eukjsdqc5c97t2zg5guqadepqqx6rc9m7wtnxy6tajjvk4a0kze4ljyuvvrpexg5up2sqxj33363v35gtew
[203, 87, 175, 176, 179, 95, 200, 156, 99, 6, 28, 153, 20, 224, 85, 0, 26, 81, 140, 117, 22]

addr1q9d66zzs27kppmxxgdzs5gp34etxla3m89g48n4z6q3m5ecw82xn7xy064e7e2zg5guqadk40nmf3057kagdzjqk7hs3840550lqg79jys0qz68rewjsqx3zc9dpzuurrhe
[19, 213, 244, 163, 254, 4, 120, 178, 36, 30, 1, 104, 227, 203, 165, 0, 26, 34, 193, 90, 17]

addr1v8ll5vf492k9g9v7ex3w32zernw535g6vsp7x7fhxpq2ptsjdemn2ve4vum8vumsxaurweng0pc8zvnsw3ekswt8ddeqj7x47t
[18, 110, 119, 53, 51, 53, 103, 54, 118, 115, 112, 55, 120, 55, 102, 104, 120, 112, 113, 50, 112, 116, 115, 104, 57, 103, 107, 114]

addr1v92fkk3qu3y68cu5ka38qhmyhx3xhxgpxqp6907m5guevlqqjd7xgj
[0]


FireCall

Link to project (GitHub): https://github.com/tylim88/FireCall
Describe the project in 2-3 sentences:  Helper Function to write easier and safer Firebase onCall function
Why would Firebase developers be interested in this project?:
FireCall standardizes how functions should handle:

Unauthorized Authentication Error
Unknown Error
Invalid Request Data Error (ZodError)
Invalid Response Data Error (ZodError)
Validate Request Data With Zod
Validate Response Data With Zod
error logging (only log the error, does not save the error into a file)

Ensuring:

always use standard HTTPS Error
end point data type safety for both ends
same function name for both ends.

Have you added a .opensource/project.json file to your site: Yes

@tylim88 Thanks! Added the project to the site, should be live within 24 hours.


KEv2: Http Proxy - AWS Signing Incorrect

What kind of request is this (question/bug/enhancement/feature request):
Bug
When attempting to make a proxy request to an Amazon endpoint the logic used to get the region and service does not take global Amazon services, like IAM, into account.
The logic here assumes host coming in is defined as https://{service}.{region}.amazonaws.com but for global services like IAM that is not the case. Note there are special cases for china and gov, see the IAM endpoints here. When the code is run through the aws signer here, region is amazonaws when it should either be an empty string or null (I am not sure of the conventions for this on the go sdk).

Verified on master-head - commit id: 6d6199121

User is able to see the KMS keys available while creating a hosted EKS cluster and while creating an AWS node template


How do i use the exporter?

Idk how to use

The exporter isn't in v2.0.0 and previous versions. If you have v2.1.0, create a folder in Wrapper: Offline called frames. Next, click on the clone option (icon that appears as two overlapping rectangular outlines), which plays the video. When the video is done playing, navigate to the frames folder and open output.mp4. It should be there, if it's not, you put it in the wrong directory.

alr, i’ll notify you when its not working.

Wait send a video about this, I wanna do it by video

I need to put the quality to high

And it also lags and skips a scene sometimes when i export it

The exporter is a new feature so there can be some bugs. The exporter does set the videos to 20 FPS though, which explains your answer. Due to its lag, it can also skip scenes. You also can't change how fast the videos export, because it is how Ffmpeg and JavaScript are encoded.

now i cant open it because the app keeps crashing, i keep reinstalling it

Well that's a problem with the JavaScript overloading your computer, because your computer is low-end, If you don't know what that means, your computer has very low RAM. Sadly it can't be fixed unless you buy more RAM, or by dusting out the interior or your computer.

i can't even do any of that

Im im re-extracting it

so why when you recorded the video player, why is it lagging constantly


sops decryption of kustomization.yml secrets from file directive

Operating system
osx ventura
Kluctl Version
latest
Kubernetes Version
v1.22
kubectl Version
v1.22
Bug description
Based on docs:

Kluctl is able to decrypt all resources referenced by Kustomize deployment items (including simple deployments ).

Kluctl shud decrypt kustomize references sops-encrypted files. (if that is missunderstanding - it would be still nice feature, even it would require specific suffix, etc..)
secretGenerator:
  - name: server-config
    files:
      - ./config/server-config.yml                # (NOT IMPLEMENTED, sops encrypted file)
  - name: certificates-from-files
    files:
      - server.crt=./config/server.crt
      - server.key=./config/server.key            # (NOT IMPLEMENTED, SOPS encrypted file)

Steps to reproduce
.sops.yaml
creation_rules:
  - encrypted_regex: '^((?i)(secret($|[^N])|values|data|stringData|username|auth|.*(token|key|private|pass|secret|domain|ssid|webhook).*))$'
    path_regex: .*.(json|yml|yaml|sops|key)
...
...

^^ same behaviour with/without path_regex
deployment.yml:
deployments:
  - path: ./views

inner kustomization.yml
components:
- ./wingmansigner

where the component ./wingmansigner/kustomization.yml looks like:
(but tested tas component, resource or even directly in above kustomization)
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

generatorOptions:
 disableNameSuffixHash: true

secretGenerator:
- name: wingman-certs
  files:
  - signer.key=./wingman.key

The file could be (yaml) or even no-yaml file:
some:
    key: aaabbbcccc
    data: |
        -----BEGIN PRIVATE KEY-----
       klsdjflksdjflksjdflsjdflasjdlfkj

but valid is non-yml structure"
-----BEGIN PRIVATE KEY-----
 klsdjflksdjflksjdflsjdflasjdlfkj

it always output in render sops encrypted string into the k8s secret.
Relevant log output
cleartext value applied/rendered in output

^^ I have added the PR to  examples, where the missing feature is better highlighted: https://github.com/kluctl/kluctl-examples/pull/12/files
basically sops decryption works for variables interpolation and for whole resources.
bug/feature would be if we could try to decrypt even other files in .build dir with pre-defined suffix, in example, these files in deployment project, that match .sops.yml: path_regex: .*.(json|yml|yaml|sops|key) ... with some default value (ie: ".secret.")

One more ifea real how to implement or deal with this he threatens back to the shebang idea if he would evaluate every file in the build a direct Tory folder first line shebang we could either Avenue eight, but the file is subs encrypted. Entendre or decrypt during that process

I'm considering changing the render phase to actually try to decrypt ALL yaml/json files. This would allow all the described use cases.
This would however make #125 an absolute requirement as otherwise people wouldn't be able to let other tools do the decryption (e.g. https://github.com/isindir/sops-secrets-operator).
What do you think?


First working version of crawler

Get the crawler running.
Make it stand-alone, specify a single project for starters.

Done


Missing items: [ITEMS]Deathfrost Boots, leggings of woven death

Links to WOTLK Wowhead for each item:

https://www.wowhead.com/wotlk/item=49890/deathfrost-boots
https://www.wowhead.com/wotlk/item=49893/sandals-of-consecration
https://www.wowhead.com/wotlk/item=49892/lightweave-leggings
https://www.wowhead.com/wotlk/item=49891/leggings-of-woven-death

these are already in


Fix broken images

Yes you are right my bad. The images don't show when previewing the markdown file on github which is where my confusion came from.


Crash in angular plugin

This is non-fatal for server, but I believe that it's crashing the angular plugin's isolate.
NoSuchMethodError: The getter 'typeParameters' was called on null.\nReceiver: null\nTried calling: typeParameters
#0      Object._noSuchMethod (dart:core-patch/object_patch.dart:42)
#1      Object.noSuchMethod (dart:core-patch/object_patch.dart:46)
#2      BindingTypeSynthesizer._instantiateClass (package:angular_analyzer_plugin/src/directive_extraction.dart:626)
#3      new BindingTypeSynthesizer (package:angular_analyzer_plugin/src/directive_extraction.dart:567)
#4      DirectiveLinker.resynthesizeDirectives (package:angular_analyzer_plugin/src/directive_linking.dart:55)

asynchronous suspension

#5      AngularDriver.resynthesizeDirectives (package:angular_analyzer_plugin/src/angular_driver.dart:700)

asynchronous suspension

#6      AngularDriver.getAngularAnnotatedClasses (package:angular_analyzer_plugin/src/angular_driver.dart:712)

asynchronous suspension

#7      AngularDriver.resolveDart (package:angular_analyzer_plugin/src/angular_driver.dart:593)

asynchronous suspension

#8      AngularDriver.pushDartErrors (package:angular_analyzer_plugin/src/angular_driver.dart:546)

asynchronous suspension

#9      AngularDriver.performWork (package:angular_analyzer_plugin/src/angular_driver.dart:237)

asynchronous suspension

#10     AnalysisDriverScheduler._run (package:analyzer/src/dart/analysis/driver.dart:1609)

asynchronous suspension

#11     AnalysisDriverScheduler.start (package:analyzer/src/dart/analysis/driver.dart:1552)
#12     new AnalysisServer (package:analysis_server/src/analysis_server.dart:383)
#13     SocketServer.createAnalysisServer (package:analysis_server/src/socket_server.dart:86)
#14     StdioAnalysisServer.serveStdio (package:analysis_server/src/server/stdio_server.dart:37)
#15     Driver.start.anonymous closure (package:analysis_server/src/server/driver.dart:434)
#16     _rootRun (dart:async/zone.dart:1120)
#17     _CustomZone.run (dart:async/zone.dart:1001)
#18     runZoned (dart:async/zone.dart:1467)
#19     Driver._captureExceptions (package:analysis_server/src/server/driver.dart:473)
#20     Driver.start (package:analysis_server/src/server/driver.dart:433)

asynchronous suspension

#22     _startIsolate.\u003canonymous closure\u003e (dart:isolate-patch/isolate_patch.dart:263)
#23     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:151)

Hmm, I haven't reproduced this one yet. I think its a race condition that won't be solved until we have an AnalysisSession. That said, I'll keep an eye out for it in my fuzzer, which gives me something reproducible.

Would it make sense to add code to guard against the exception? Or code to produce some debugging output that you could use to find and fix the underlying problem? Based on internal logs, this appears to be happening not infrequently.


🚀 Feature: Bedrock Instrumentation

Which component is this feature for?
All Packages
🔖 Feature description
Instrument calls to AWS's [Bedrock(https://www.npmjs.com/package/@aws-sdk/client-bedrock-runtime), including adding attributes for input parameters, model, etc. - similarly to our OpenAI instrumentation. The instrumentation should support all types of calls - streaming, non streaming, async, etc.
Instrumentation should support all models supported in our Python instrumentation.
This should specifically work with Google's new Gemini model.
🎤 Why is this feature needed ?
Completeness of OpenLLMetry
✌ How do you aim to achieve this?
Similar to other instrumentations we have in this repo.
🔄 Additional Information

Make sure to initialize this in Traceloop SDK.
Add proper tests.
Add a sample in the sample app.
Upload a screenshot from Traceloop with how your instrumentation works.

👀 Have you spent some time to check if this feature request has been raised before?

[X] I checked and didn't find similar issue

Are you willing to submit PR?
None

/bounty $100

/attempt #33

/attempt #33

/attempt https://github.com/traceloop/openllmetry-js/issues/33


Circular reference internal typealias Model Xcode 11.4

Using Xcode 11.4, and SIdeMenu from SPM, you cannot build and receive this error
Circular reference
On this line
   internal typealias Model = MenuModel & PresentationModel & AnimationModel

There was a pull request that solved this. My bad

Same error occurs also with CocoaPods instead of SPM

Will there be a cocoapods release to fix this in the near future?
Cheers

Same on Xcode 11.4 beta 3 (11N132i) on the Cocoapods release.


New 7.6 user/perm screen improvements

[x] Doc perms modal dialog - Show inherited perms (similar to how app perms shows them), e.g. Sub Sub Group is a member of Sub Group, so it shows perms inherited from Sub Group.

[x] Action button column - On the various perms screen tables it would be good to have an action button column (elipses icon) to do actions on the row, e.g. on UserGroups, click on a group then right click to edit the app perms (or goto the app perms scrn).  Essentially make it easier for the user to jump to a user/group/permission/etc.

[x] Change the App Perms scrn should default to showing all users/grps, not just those with app perms.  E.g. I just created a grp and want to give it app perms.  Go to scrn and can't see it, not obv why.  Same for the doc perms modal.

[ ] Change the Members in Group pane to optionally show indirect members rather than just the direct members.  Probably needs a column to show which group(s) each member directly belongs to.  Ability to jump to any of those groups also beneficial.

[ ] Decide how the QuickFilter and expression tree filter should be presented to the user as they do essentially the same thing but in a different way.  The former is better for quickly filtering a list to find something, but the latter is better for doing a complex batch update.  Maybe the batch change should be a screen that includes the filter icon, the filtered table and the changes to be made.

[ ] Create a PathCell so we can specifically render an explorer tree path (i.e. System / Folder A / Folder B), e.g. making the / a different colour and making each path part a clickable thing (either to locate that node in the tree or open the folder).  Applies to User Perms Report screen for one.

[ ] On the User Perms Report screen replace the eye icon with a drop down to toggle between showing all docs, only those with explicit perms, only those with inherited perms and ones with inherited+explicit.

[ ] Change the User Perms Report screen to sort in a case insensitive way.

Closing as I have copied all these tasks into #4594


Add configuration option to not collect data that requires installer or DIY account

This option allows to not collect data that requires authorization through a installer or DIY account. Enabling the option will prevent repeated new enphase token collection and still have resulting 401 status from the envoy as latest firmware does not allow home owner account access to installer data.
When the integration is configured go to the configuration options, check the 'Do not collect data that requires installer or DIY enphase account' option and reload the integration.

Hi @catsmanac. Thanks for your PR. Can you please reformat the files you changed with black? This ensures we have uniform code formatting.


Disable prop for button doesn't change color

I want to use the disable prop in my button component. It works fine, it change the color to grey. But when i add background color to the button, the disable prop works but it doesn't change the color to grey.
                {/* <Button style={styles.mainBtn}>
                    <Text style={styles.btnText}>Submit</Text>
                </Button> */}

// styles
mainBtn: {
backgroundColor: '#0C1248',
borderRadius: 20
},
btnText: {
color: 'white',
fontSize: 18,
},

@zaid0077 That's how it is supposed to work. If you specify a background color, it will take that as the background color in priority.
If you want the button to be grey when disabled and applied color when not disabled, then you can pass the background color conditionally.
Let us know if you need further help with this.

@zaid0077 That's how it is supposed to work. If you specify a background color, it will take that as the background color in priority.
If you want the button to be grey when disabled and applied color when not disabled, then you can pass the background color conditionally.
Let us know if you need further help with this.

facing the same issue

@jimbo23
Use isDisabled instead of disable props. And,  Put the string directly under the Button without using Text.
<Button isDisabled={!email || !password} >Sign In</Button>

None of you address the original issue which is that having the button disabled overrides the custom background color if we pass one to the button.
i.e. We set the button's background color to red, if the button is disabled we still expect the background color to be red but it's not the case anymore.

Have same issue here. _disabled props doesn't work

same issue here

@jingcheng16 did you found any solution?

Have same issue here. _disabled props doesn't work

@jingcheng16 did you found any solution?

Have same issue here. _disabled props doesn't work

@Irfanwani Yes I solved it. Try isDisabled, let me know if it works.

@jingcheng16 you are right, buttons are dimmed a bit when disabled which is pretty nice but I just wanted some more customisations. Anyway, going with the default behavior now.


[python] NumPy 2 experiment [WIP]

Issue and/or context: #2536
Changes:
Notes for Reviewer:
Main thing to watch for: NumPy 2.0 does not support Python 3.8, while we have not yet dropped support for Python 3.8 (#2141).
Per https://github.com/numpy/numpy/releases/tag/v2.0.0:

The Python versions supported by this release are 3.9-3.12.

I thought we'd need a bit in setup.py like
        # There is no Python 3.8 support for NumPy 2
        "numpy<2; python_version<'3.9'",
        "numpy; python_version>='3.9'",

but in hindsight it seems not -- a Python 3.8 asking to get NumPy will simply not be offered version 2 in the first place 😎

Oddly, the "Show package versions" bit in
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/10585607673/job/29332572332?pr=2934
is showing even for Python 3.11
Run python scripts/show-versions.py
tiledbsoma.__version__    1.12.0rc0.post106.dev3370075614
tiledb.version()          0.31.1
core version              2.25.0
anndata.__version__  (ad) 0.10.8
numpy.__version__    (np) 1.26.4
pandas.__version__   (pd) 2.2.2
pyarrow.__version__  (pa) 17.0.0
scanpy.__version__   (sc) 1.10.2
scipy.__version__    (sp) 1.14.1
python__version__         3.[11](https://github.com/single-cell-data/TileDB-SOMA/actions/runs/10585607673/job/29332572332?pr=2934#step:9:12).9

🤔

I just pushed a rebase. Good news: this time (unlike before) we do get numpy 2
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/11347335957/job/31558479564?pr=2934
python scripts/show-versions.py
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/[3](https://github.com/single-cell-data/TileDB-SOMA/actions/runs/11347335957/job/31558479564?pr=2934#step:9:3).9.20/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.9.20/x6[4](https://github.com/single-cell-data/TileDB-SOMA/actions/runs/11347335957/job/31558479564?pr=2934#step:9:4)/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.20/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.20/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.20/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.20/x64/lib
tiledbsoma.__version__    1.1[5](https://github.com/single-cell-data/TileDB-SOMA/actions/runs/11347335957/job/31558479564?pr=2934#step:9:5).0rc0.post28.dev4289275158
tiledb.version()          0.32.3
core version              2.2[6](https://github.com/single-cell-data/TileDB-SOMA/actions/runs/11347335957/job/31558479564?pr=2934#step:9:6).2
anndata.__version__  (ad) 0.10.9
numpy.__version__    (np) 2.0.2
pandas.__version__   (pd) 2.2.3
pyarrow.__version__  (pa) 1[7](https://github.com/single-cell-data/TileDB-SOMA/actions/runs/11347335957/job/31558479564?pr=2934#step:9:7).0.0
scanpy.__version__   (sc) 1.10.3
scipy.__version__    (sp) 1.13.1
python__version__         3.9.20

Trying to add @kounelisagis as a reviewer ...


bookmark always returns null

Hi,
so following Part 6 got me to my first error I couldn't solve yet.
In my BookmarksController.js my bookmark will always be null.
const {songID, userID} = req.query;
    const bookmark = await Bookmark.findOne({
           where: {
               SongId: songID,
               UserId: userID
           }
       });
res.send(bookmark || {});

And my bookmark.json file looks like this:
[
  {
    "SongId": 1,
    "UserId": 1
  }
]

Can you maybe have a short look on the repo or do you have any clue why it does not find anything? https://github.com/akratzel/guitar-tabs/tree/dev
Maybe a sequelize version issue?
Thanks!

i've the same issue, you solved?

For me it worked I changed
const {songID, userID} = req.query;
to
 const songId = req.body.params.songId    const userId = req.body.params.userId

Have you solved the issue? My query just take the songId but not the userId so it always gives me 500 error. Tried few things, but none worked..


Improve odo watch integration test

[kind/Enhancement]

Which functionality do you think we should update/improve?
Only one watch test scenario https://github.com/openshift/odo/blob/master/tests/integration/watch_test.go#L54 has been running since a long period of time. So more scenario can be added using the combination of flags.
$ odo watch -h
Watch for changes, update component on change.

Usage:
  odo watch [component name] [flags]

Examples:
  # Watch for changes in directory for current component
  odo watch

  # Watch for changes in directory for component called frontend
  odo watch frontend

Flags:
      --app string           Application, defaults to active application
      --context string       Use given context directory as a source for component settings
      --delay int            Time in seconds between a detection of code change and push.delay=0 means changes will be pushed as soon as they are detected which can cause performance issues (default 1)
  -h, --help                 Help for watch
      --ignore stringSlice   Files or folders to be ignored via glob expressions.
      --project string       Project, defaults to active project
      --show-log             If enabled, logs will be shown when built

Global Flags:
      --skip-connection-check   Skip cluster check
  -v, --v Level                 Log level for V logs. Level varies from 0 to 9 (default 0).
      --vmodule moduleSpec      Comma-separated list of pattern=N settings for file-filtered logging

Why is this needed?
To add more integration test coverage

/assign @prietyc123

Please describe the scenarios in the issue before implementing it.
My take on this is that it is far more important to test the watch with different images than go overboard with testing every single flag that command has.

Let's start with having at least one test for each flag, no need to start with permutations.

We have good watch tests already and there are issues related to improving intg tests which covers this if need be


excludedTargets parameter default value problem

From 1.2.0 there's a new xcodeBuildCoverage method implemented in #26 that allows using [ExcludedTarget] type for excludedTargets parameter.
But the problem is, in that method the excludedTargets parameter also has a default value [], which causes a big problem that for projects that don't have excluded targets, there's no way for the compiler to check which method to use, the excludedTargets: [String] one or the excludedTargets: [ExcludedTarget] one. And that will cause a build error:
/var/folders/6q/wgy6jtp12w5gzgm9lzcglpqw0000gn/T/_tmp_dangerfile.swift:577:1: error: ambiguous use of 'xcodeBuildCoverage(_:minimumCoverage:excludedTargets:hideProjectCoverage:)'
Coverage.xcodeBuildCoverage(.xcresultBundle(ProcessInfo.xcTestResultPath), minimumCoverage: 60)
^
/Users/vagrant/.danger-swift/Scripts/Cache/-Users-vagrant-git-_dangerfile_imports/.build/checkouts/danger-swift-coverage/Sources/DangerSwiftCoverage/DangerSwiftCoverage.swift:19:24: note: found this candidate
    public static func xcodeBuildCoverage(_ coveragePathType: CoveragePathType, minimumCoverage: Float, excludedTargets: [String] = [], hideProjectCoverage: Bool = false) {
                       ^
/Users/vagrant/.danger-swift/Scripts/Cache/-Users-vagrant-git-_dangerfile_imports/.build/checkouts/danger-swift-coverage/Sources/DangerSwiftCoverage/DangerSwiftCoverage.swift:24:24: note: found this candidate
    public static func xcodeBuildCoverage(_ coveragePathType: CoveragePathType, minimumCoverage: Float, excludedTargets: [ExcludedTarget] = [], hideProjectCoverage: Bool = false) {

I think the best way to solve this problem is to remove the default value for [ExcludedTarget] method so for the projects that don't have an excludedTargets parameter, the build can still be passed

Oh I didn't see this coming, nice find.

Thank you, https://github.com/f-meloni/danger-swift-coverage/pull/28 should fix it


PHP7 support

Thanks for the PR! Unfortunately, the build is now failing because coverallis seems to require 5.5+ but the library supports 5.3+. If you can find a way to get the build to pass with 5.3+ (probably pin the coverallis version), I can merge the request.

@archwisp We should use "satooshi/php-coveralls": "~1.0" since it supports PHP 5.3, not dev-master. Any reason to use dev-master?
I tried to use "satooshi/php-coveralls": "~1.0", but I got:
$ composer update satooshi/php-coveralls
You are running composer with xdebug enabled. This has a major impact on runtime performance. See https://getcomposer.org/xdebug
Loading composer repositories with package information
Updating dependencies (including require-dev)
  - Installing symfony/yaml (v2.8.2)
    Downloading: 100%

  - Installing symfony/stopwatch (v3.0.1)
    Loading from cache

  - Installing symfony/polyfill-mbstring (v1.0.1)
    Loading from cache

  - Installing symfony/console (v3.0.1)
    Loading from cache

  - Installing symfony/filesystem (v3.0.1)
    Loading from cache

  - Installing symfony/config (v3.0.1)
    Downloading: 100%

  - Installing psr/log (1.0.0)
    Loading from cache

  - Installing symfony/event-dispatcher (v2.8.2)
    Downloading: 100%

  - Installing guzzle/guzzle (v3.9.3)
    Loading from cache

  - Installing satooshi/php-coveralls (v1.0.0)
    Downloading: 100%

  - Installing phpunit/php-text-template (1.2.1)
    Loading from cache

  - Installing phpunit/phpunit-mock-objects (1.2.3)
    Loading from cache

  - Installing phpunit/php-timer (1.0.7)
    Loading from cache

  - Installing phpunit/php-token-stream (1.2.2)
    Loading from cache

  - Installing phpunit/php-file-iterator (1.4.1)
    Loading from cache

  - Installing phpunit/php-code-coverage (1.2.18)
    Downloading: 100%

  - Installing phpunit/phpunit (3.7.38)
    Downloading: 100%

symfony/console suggests installing symfony/process ()
symfony/event-dispatcher suggests installing symfony/dependency-injection ()
symfony/event-dispatcher suggests installing symfony/http-kernel ()
guzzle/guzzle suggests installing guzzlehttp/guzzle (Guzzle 5 has moved to a new package name. The package you have installed, Guzzle 3, is deprecated.)
satooshi/php-coveralls suggests installing symfony/http-kernel (Allows Symfony integration)
phpunit/phpunit suggests installing phpunit/php-invoker (~1.1)
Writing lock file
Generating autoload files
Warning: Ambiguous class resolution, "PHPCS_Sniffs_ControlStructures_ControlSignatureSniff" was found in both "vendor/phpunit/phpunit/build/PHPCS/Sniffs/ControlStructures/ControlSignatureSniff.php" and "vendor/phpunit/php-token-stream/build/PHPCS/Sniffs/ControlStructures/ControlSignatureSniff.php", the first will be used.
Warning: Ambiguous class resolution, "PHPCS_Sniffs_Whitespace_ConcatenationSpacingSniff" was found in both "vendor/phpunit/phpunit/build/PHPCS/Sniffs/Whitespace/ConcatenationSpacingSniff.php" and "vendor/phpunit/php-token-stream/build/PHPCS/Sniffs/Whitespace/ConcatenationSpacingSniff.php", the first will be used.
Warning: Ambiguous class resolution, "PHPCS_Sniffs_ControlStructures_ControlSignatureSniff" was found in both "vendor/phpunit/phpunit/build/PHPCS/Sniffs/ControlStructures/ControlSignatureSniff.php" and "vendor/phpunit/php-code-coverage/build/PHPCS/Sniffs/ControlStructures/ControlSignatureSniff.php", the first will be used.
Warning: Ambiguous class resolution, "PHPCS_Sniffs_Whitespace_ConcatenationSpacingSniff" was found in both "vendor/phpunit/phpunit/build/PHPCS/Sniffs/Whitespace/ConcatenationSpacingSniff.php" and "vendor/phpunit/php-code-coverage/build/PHPCS/Sniffs/Whitespace/ConcatenationSpacingSniff.php", the first will be used.
Warning: Ambiguous class resolution, "PHPCS_Sniffs_ControlStructures_ControlSignatureSniff" was found in both "vendor/phpunit/phpunit/build/PHPCS/Sniffs/ControlStructures/ControlSignatureSniff.php" and "vendor/phpunit/phpunit-mock-objects/build/PHPCS/Sniffs/ControlStructures/ControlSignatureSniff.php", the first will be used.
Warning: Ambiguous class resolution, "PHPCS_Sniffs_Whitespace_ConcatenationSpacingSniff" was found in both "vendor/phpunit/phpunit/build/PHPCS/Sniffs/Whitespace/ConcatenationSpacingSniff.php" and "vendor/phpunit/phpunit-mock-objects/build/PHPCS/Sniffs/Whitespace/ConcatenationSpacingSniff.php", the first will be used.

Why do we need "classmap": [""],?

@umpirsky Thanks for the PR! I opened the following PRs to fix the issues you found:
#19 - Remove classmap from Composer config
#20 - Pin satooshi/php-coveralls to PHP 5.3+ compatible version
Once those are merged we can rebase against master to see your PR's updated Travis runs.

@johnkary Wow, that was fast! :trollface:
However, better ever then never. :)


Alias storage test and enhance parser

Pull Request Test Coverage Report for Build 430

13 of 21 (61.9%) changed or added relevant lines in 8 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.5%) to 82.837%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/main/java/seedu/address/model/alias/UniqueAliasList.java
0
1
0.0%

src/main/java/seedu/address/logic/parser/AliasCommandParser.java
1
2
50.0%

src/main/java/seedu/address/model/ModelManager.java
0
1
0.0%

src/main/java/seedu/address/model/AddressBook.java
0
1
0.0%

src/main/java/seedu/address/logic/commands/AliasCommand.java
1
5
20.0%

Totals

Change from base Build 429:
0.5%

Covered Lines:
2669

Relevant Lines:
3222

💛  - Coveralls


If elapsed time is more than  800ms clock icon becomes red, can someone hep me ti fix this

There's really nothing to fix. This is how the reporter works. You can tweak what mocha considers a "slow" test. https://mochajs.org/#-s---slow-ms

Thanks, it worked.


Bug_FormFieldContainer> Default> 'isInverse' control prop> Text 'Hello test' gets disappear if 'isInverse' control prop is 'True'.

Describe the bug
Text 'Hello test' gets disappear if 'isInverse' control prop is 'True'.
To Reproduce
Steps to reproduce the behavior:

Go to 'https://storybook-preview-290--upbeat-sinoussi-f675aa.netlify.app/?path=/story/formfieldcontainer--default'
Click on 'Controls'
Select the 'isInverse' control prop
Toggle it into 'True'
Verify the preview

Expected behavior
Dark background should appear if 'isInverse' control prop is 'True'.
Screenshots
https://somup.com/c3nDi9TeTO
Desktop:
OS: [Win 10]
Browser [chrome]
Version [Version 98.0.4758.82 (Official Build) (64-bit)]
Additional context

As per the documentation, If 'isInverse' is true, the component will have inverse styling to better appear on a dark background. However, for 'FormFieldContainer', text gets disappears instead of showing the dark background.

Now that we've added an addon to Storybook to see a dark background, we can confirm that flipping isInverse to true does not make the text disappear, it's just white (and invisible against the default white background):
https://storybook-preview-636--upbeat-sinoussi-f675aa.netlify.app/?path=/story/formfieldcontainer--default&args=isInverse:true&globals=backgrounds.grid:false;backgrounds.value:!hex(333333)
The consumer using this component would be providing the dark background (if isInverse is true) to wrap this component, so I don't think it's necessary at this time for us to add that here. Closing as won't fix.


[KARAF-5667] Installing the audit-log feature never ends

Please retest this.

retest this please


Documentation: Rename example for env file

The readme suggests the following:
Add a .gitlab-ci-local-env file to the current working directory or a .env file in $HOME/.gitlab-ci-local.
According to gcl --help though, the default name is .gitlab-ci-local-variables.yml.
It took me some time to figure out why my variables were not accessible inside my executor's docker containers.
So I would suggest to make the documentation a bit more self-explanatory like this:
Add a .gitlab-ci-local-variables.yml file to the current working directory or a .env file in $HOME/.gitlab-ci-local..
Thanks in advance!

This is frequently confused.
This https://github.com/firecow/gitlab-ci-local?tab=readme-ov-file#dotenv-file section of the docs is referring to the following

For what you are trying to do, you should actually refer to:
https://github.com/firecow/gitlab-ci-local?tab=readme-ov-file#home-file-variables
https://github.com/firecow/gitlab-ci-local?tab=readme-ov-file#project-file-variables
Hopefully this is clearer now.  Feel free to open an mr to phrase it better

fixed in https://github.com/firecow/gitlab-ci-local/pull/1434


Suggestion: please give docker pull a "dry -run" option

Hi,
I have been trying to write a script to check for updates, but this has proven to be extremely cumbersome. A simple way to this would be if [docker pull]  would have a flag like [C, just checking] that says [dont bother] if no upstream update has arrived.
I see lots of folks online trying to check for upstream updates: this seems much to complicated now. Or have a missed something and is there an easy way: please enlighten me then!
My scripting tries to:

check what digests images I am using locally in containers have (not difficult)
check if upstream images have been updated by comparing their digests to locally used digests (havent found a way yet to find digest of latest upstream image)

The aim is not to pull images if not needed and if needed, to script local containers to be built/run again with the updated upstream image.
Loving docker by the way: what a speed to built stuff!

This would really help in an CI-environment with a private registry (e.g. Gitlab). The script can check if the tag exists. If not, build it and upload it. Otherwise just skip the step.

@hanscees You can use a script like this to validate if manifest exists before pulling.
#!/bin/bash
# call it as <script> company/repo manifests/{digest|tag}
repo=$1
url=$2
token=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$repo:pull" | jq -r .token)
curl https://registry-1.docker.io/v2/$repo/$url -X HEAD -H "Authorization: Bearer $token" -L -H "Accept: application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.list.v2+json"

Note that script talks to Docker Hub registry but can be easily modified for other registry also.
However I am still confused about your requirement. When you base your image on a tag and if tag gets updated then your image should also get updated. If the tag is not updated and the image exists locally then the image shouldn't be built. Is the compose file setup or something else?

Just here to +1 this. I have the same issue where I'd like to ensure that all images/tags actually exist before building/publishing my docker-compose.yml file. Obviously, I can just "docker-compose pull", but that can be slow and can also suffer from other issues that have nothing to do with the docker-compose file (i.e, non-existent env_files on the build agent, etc), not to mention being unnecessarily slow and disk-space consuming.

+1

+1

+1

+1

can't believe it - it's been years for something as trivial as "check if remote repository image exists, if yes - print digest if no - give error" - there is no native way to just check the remote repo, very frustrating docker experience... It is a good thing it will be tool of the past, after not long time.

@voyager123bg what about
docker buildx imagetools inspect debian:latest?
or
docker manifest inspect ruby ?
^ "Check if a remote repository exists, if yes - print all sort of info including digest if no, give error"

can't believe it - it's been years
there is no native way to just check the remote repo

Looks like there is (two) way(s) (probably since at least 19.03 - over two years now).
Furthermore, a registry just exposes a plain http API - a little bit of bash and curl would give you exactly the same thing very fast (authentication is the only mildly tricky part if the service uses something else than basic) though @manishtomar example above https://github.com/distribution/distribution/issues/2412#issuecomment-422877499 is literally one line.

very frustrating docker experience... It is a good thing it will be tool of the past, after not long time.

Chill, it's just software.
Also, what happened to reading the documentation before losing your cool? Dog ate your laptop? :-)
@thaJeztah this ticket should be closed (docker manifest inspect should let the OP do whatever they need IMHO).

@voyager123bg: See https://stackoverflow.com/questions/32113330/ for snippts
@dubo-dubon-duponey: A workaround is not a fix! And if it's that trivial - why not just add a dry-run option?
And your API-Solution boils down to "Write your own client", as things quickly become a bit more complicated once you need authentication - nothing you can't fix with a bash-script, but nonetheless.

The registry API already provides methods/endpoints to check if an image exists and get the digest.
If using the API is not an option and you want your client app to do it instead, then please raise an issue for that client app. If it’s Docker CLI then raise an issue in https://github.com/docker/cli.
Closing this issue.

@dubo-dubon-duponey: A workaround is not a fix!

Beside using the API, the feature requested by the OP does already exist in the docker cli , as pointed out above: this is the manifest command.
Here is how you get the digest of a specific remote image for a specific architecture:
docker manifest inspect "$image" | jq -rc '.manifests[] | select(.platform.architecture=="'$arch'") | .digest'

And if it's that trivial why not just add a dry-run option?

@joaodrp just addressed that question.
Other clients - like terraform-provider-docker for example - do provide some form of that (for terraform, the --plan option is much more powerful than just checking whether digests have been updated). Compose does not seem to have such an option (last I checked), but I'm sure there is an open issue there (and given compose scope, that would make sense).
Hope that helps.

Just landed here and was excited, but want to point out not all registries are supported by docker manifest inspect:
$ docker manifest inspect nvcr.io/nvidia/cuda:11.8.0-runtime-ubuntu20.04
failed to configure transport: unsupported registry version

+1

If you want to get a raw manifest, skopeo or podman are good options.
jammy:~$ podman manifest inspect nvcr.io/nvidia/cuda:11.8.0-runtime-ubuntu20.04
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "manifests": [
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 2205,
            "digest": "sha256:872f03ad4329a99c7c9812657c6ee1a867d779a9887af93ae810a7bfa45d69ca",
            "platform": {
                "architecture": "arm64",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 2207,
            "digest": "sha256:bc7784b3314eda17950a21417810caafbd5f7e0844c7d3ec215208128e69f8ce",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        }
    ]
}

jammy:~$ skopeo inspect --raw docker://nvcr.io/nvidia/cuda:11.8.0-runtime-ubuntu20.04
{
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "schemaVersion": 2,
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:872f03ad4329a99c7c9812657c6ee1a867d779a9887af93ae810a7bfa45d69ca",
         "size": 2205,
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:bc7784b3314eda17950a21417810caafbd5f7e0844c7d3ec215208128e69f8ce",
         "size": 2207,
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }
   ]
}


Demo

Description 📜
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context.
Fixes # <issue_number_here>

Type of change 📝

[ ] Audio (Should be in mp3 format Includes speech clarity, Concise ,Low distortion)
[ ] Video (Animations, screen-recordings, presentations and regular explanatory films are all possibilties etc)
[ ] Documentation (Content Creation in the form of codes or tutorials)
[ ] Other (If you choose other, Please mention changes below)

Domain of Contribution 📊

[ ] Android Dev (Flutter, Java, Kotlin)
[ ] Backend Dev (Java, .NET, Python)
[ ] C/CPP
[ ] Competitive Programming
[ ] Cyber Security
[ ] DSA
[ ] Database
[ ] Datascience with Python
[ ] Datascience with R
[ ] Frontend Dev (HTML, CSS, JS)
[ ] Frontend Dev (React, Angular, Vue)
[ ] Golang
[ ] Interview Prep
[ ] Java
[ ] JavaScript
[ ] MERN
[ ] Machine Learning
[ ] Open Source
[ ] Python
[ ] Rust
[ ] Statistics
[ ] UI/UX

Checklist ✅

[ ] I follow Contributing Guidelines & Code of conduct of this project.
[ ] I have performed a self-review of my own code or work.
[ ] I have commented my code, particularly in hard-to-understand areas.
[ ] My changes generates no new warnings.
[ ] I'm GWOC'21 contributor

Screenshots / Gif (Optional) 📸

@luvksahu You Java branch is not even with our Java branch. Please resolve this. If you are not able to do that, tag me in discord.


Fix a few bugs and add cool things

The commits names literally explain most of the time what I did.

Hold on
I wanna try to add Actions builds on my phone

yipee now it's done!!
@YoshiCrafter29 if you see this, please, can you merge this or tell me if there's something wrong?

i don't think i'll merge this commit
the issue with is is the syntax "fixing"
i prefer keeping it that way since i have 100% control over it
that pr looked good but im sorry i can't merge it.

i don't think i'll merge this commit
the issue with is is the syntax "fixing"
i prefer keeping it that way since i have 100% control over it
that pr looked good but im sorry i can't merge it.

thanks for the feedback
i understand about the syntax stuff and i would really remove that, but i can't revert it (i use this to do it)


Enable SQLITE_ENABLE_DBSTAT_VTAB

PR for https://github.com/simolus3/sqlite3.dart/issues/237

@simolus3 looks like maven is updated now. I upgraded the native library in the grade file as well now.


Apps take a long time to start or don't at all (sometimes)

Seen this in a few occasions. Every now and then apps (ie browser, sailotp from the store) stay in the cover view with the spinning circle and either take some time to move to full screen view or disappear.
I run a free -h in the terminal after it happened and didn't notice anything weird -at least to my untrained eye.

Hard to say, maybe RAM allocation related. I will ask few questions around regarding it

I got into the same problem while testing battery consumption, i.e. everything is off, and waking up phone every hour to open terminal app and log stats.
in few cases, starting the terminal app takes some time, and in my last "logging" attempt the terminal app took too long to start it disappeared from the app tray. Second attempt to start the terminal was fast.

One note on the behavior. When an app gets stuck if you open another app, minimize it, and tap the stuck app it usually starts.
Just writing this in case it helps finding the cause of the issue.

I've encountered the same issue several times, the system is not busy, there is plenty of ram available, is there any way to monitor what is causing that?
e.g. I had by chance an ssh session open, and while starting the browser spinning, I run free, w, and top
[nemo@Sailfish Documents]$ free
              total        used        free      shared  buff/cache   available
Mem:        3765028     1757500     1081268       32332      926260     2341380
Swap:       1048572           0     1048572
[nemo@Sailfish Documents]$ w
 11:03:45 up 10:26,  1 user,  load average: 2.18, 2.39, 2.67
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
top - 11:04:02 up 10:26,  1 user,  load average: 2.25, 2.40, 2.68
Tasks: 700 total,   1 running, 699 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.4 us, 25.8 sy,  0.2 ni, 67.2 id,  0.1 wa,  0.6 hi,  0.7 si,  0.0 st
MiB Mem :   3676.8 total,   1056.6 free,   1715.1 used,    905.0 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.   2287.8 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
25090 root      20   0       0      0      0 S   2.0   0.0   2:43.19 kworker/u16:17
31320 root      20   0       0      0      0 S   1.7   0.0   0:42.66 kworker/u16:15
31670 nemo      20   0    6972   3236   2192 R   1.0   0.1   0:00.25 top
30351 root      20   0       0      0      0 S   0.5   0.0   0:00.07 kworker/u16:29
  692 root      20   0    4508   2524   2096 S   0.4   0.1   2:17.69 systemd-journ+
  720 root      20   0   11952   3696   2448 S   0.3   0.1   2:11.42 systemd-udevd
30148 root      20   0       0      0      0 S   0.2   0.0   0:00.09 kworker/u16:12
    7 root      -2   0       0      0      0 S   0.2   0.0   0:25.72 rcu_preempt
   15 root      rt   0       0      0      0 S   0.2   0.0   0:38.90 migration/0
   28 root      rt   0       0      0      0 S   0.2   0.0   0:29.21 migration/2
   46 root      rt   0       0      0      0 S   0.2   0.0   0:20.54 migration/4
 3190 root      20   0   20944   8892   7780 S   0.2   0.2   1:14.77 mce
 3428 root      20   0   32116   7684   6336 S   0.2   0.2   1:02.37 connmand
 6497 root      20   0       0      0      0 S   0.2   0.0   0:04.56 kworker/2:6
    8 root      -2   0       0      0      0 S   0.1   0.0   0:13.53 rcu_sched
   11 root      20   0       0      0      0 S   0.1   0.0   0:06.78 rcuos/0
   18 root      20   0       0      0      0 S   0.1   0.0   0:20.87 cpuhp/1
   27 root      20   0       0      0      0 S   0.1   0.0   0:14.44 cpuhp/2
   33 root      20   0       0      0      0 S   0.1   0.0   0:11.84 rcuop/2
   34 root      20   0       0      0      0 S   0.1   0.0   0:05.02 rcuos/2

when top started there was a kworker process at 50% cpu

I don't know and nor have any theory to test regarding this slow start and sometimes non-starting. You could try to look into logs (journalctl) and see whether there is some anomaly at that time. As for RAM allocation mysteries, I will still have to ask around - looks like in Jan 2020 though as I will have rather limited amount of time before that.

No worries, I've attached the output of journalctl and dmesg logs for a reference.
journalctl.log
dmesg.log

For reference, indicate also time when it happened. As going through MBs of logs without a time reference makes it rather difficult. :)

deleted previous comment and renamed logs to include timestamp and model
journalctl_xz3_28_12_2019.log
dmesg_xz3_28_12_2019.log

But we need also to know when app starting issue happened, as precisely as possible.

https://together.jolla.com/question/219961/xa2-bad-memory-handling-lots-of-oom-killing/
I am putting this here also since it seems to be similar. Its weird that it seems to happen with phones that have quite a bit of ram and ie my J1 never had it.

Something weird happened today. I went to start an app and the phone interface "froze". With a long press at the power button i managed to get it to shutdown.
Could it be the case we have some memory handling issues or leaks?? With free -h i've noticed the free memory dropping from 2.2G (after a reboot) to 1G after a few days of uptime.

three comments:

I experience on regular basis apps failing to start with first tap, usually 2nd attempt apps start fast, also some times it takes long time to wake up screen, e.g. 2+ sec
a week ago the phone could not wake up, instead the spinning circle was displayed, long scroll from top the settings menu was accessible, I was also able to take pictures, but nothing more as the device was locked. Unfortunately remote access was disabled and the only option i had to regain access was to reboot the phone.
free memory is not always a safe indication on its own.

Frozen phone: you may try to restart lipstick. I think it was double click on power button - not a simple solution but may work. I have used it once in a while.
Apps not starting - occasionally happens to me as well. But mostly due to the bugs in the programs that I develop. Will have to catch logs when its something else.
And with free memory on Tama - will have to look into it.

Things i observed.
I run:
ps -o pid,user,%mem,command ax | sort -b -k3
The process that consumes most of the memory -from restart- is lipstick and seems to be going up as days pass without a reboot. So its either lipstick that causes the stuck interface issue (although it has memory and swap space to do its stuff) or something that doesn't gets listed with the above command.

Interesting. How large is Lipstick memory consumption for you? For me, RSS is about 150MB, which is not that much. Uptime 5 days

I 've only seen the percentages from the command above. starts at 3.7 and goes up to 4.3 as days go by.

That's similar over here. Don't think 4% would be causing OOM killer

And i suspect neither the freezes also. So its probably something else.

@rinigus one thing though. When an app is stuck at the minimized state (before it goes full screen) it -usually- goes to full screen if you tap it again. So it might not be a memory problem but something that happens before the transition to full sceen. What i have no idea.

Correct, and if you do not wait for the app to be killed, you can starg a second instane of tge app without issues. Sometimes i notice a similar behaviour waking up the screen, it takes a couple of seconds to wake up.
My feeling is that the app somehow has a very slow start up and get killed by another monitoring process  as it's slow to respond? Very occasionaly similar behaviour is observable in other officially supported devices.
--
Sent from my Jolla

That I have seen (clicking on minipage with delayed start), mainly on Flatpak apps. Although nothing alarming found in the logs. Could be some Lipstick interaction with an app starting slowly for some reason. Although, Flatpak Runner is a small app by itself...

Most likely -or definately- lipstick needs a complete overhaul. Comply to the latest protocol and fix performance issues, if those are caused by it. Its not reasonable to have SFOS running on phones that can run fortnite and SFOS to have issues.

This is not limited to Lipstick. I have just had Flatpak Runner started and Angelfish failing to do so. After reboot, all was fine.
Didn't debug it further as I was finishing few things with Angelfish development and didn't have time for it. But in this respect, Lipstick could be just a "messenger"

@rinigus i think this needs to be brought up in a mailing list.  :man_shrugging:

Maybe. But I would prefer to be better prepared before asking for help. Will grab some logs next time, silly that I didn't do it this time.

After observing this for a while i am sure what eats the memory isn't something that sill show up with the ps command above.

I have the similar feeling, based on collectd data. Something doesn't add up, but have to figure out what.

@rinigus can you test something if you have time. It seems to me that things get worst in regards to memory (and eventual freeze) when streaming audio to a device.
Might worth trying if possible.

Someone suggested today (in the meeting) that i should test with starting the apps from the terminal and see if one gets stuck. After a lot of attempts nothing out of the ordinary happened. All apps started normally. So something happens whey you start the apps from the app tray.
Just mentioning this here. If someone else can test feel free and mention your results.

I doubt it will be observed from a terminal, the only way to get info is indirectly from logs?  hopefully upgrade to 3.3 will help?
--
Sent from my Jolla

Terminal bypasses Sailfish boosters. I have seen on my earlier devices that sometimes my own app doesn't start from GUI but runs fine from terminal. So, I doubt as well that the terminal will help much.
As for update, will work on it when we get target at OBS - it's still absent so far. But it will come for sure.

is there any way to log what is going on with the boosters??

I have not noticed long app start times since 3.3 upgrade, and also I have the feeling battery consumption is better

@pagism i just logged in to comment on that. I haven't noticed the issue in 3.3 and was about to ask if anyone is experiencing the same.

We have just rebooted recently, let's wait whether it will resurface with time. But there were plenty of kernel updates included this time as we had a longer break between releases. Let's hope that it got resolved by great work of Sony AOSP team that they do on the kernels.

I am closing this since it seems that the issue with apps taking a long time to start be a thing of the past. I had a reboot or two nothing "extreme" like the previous release.
There still seems to be something that makes the free memory reduce over time (checking with free -h) but overall the phone is much more stable
In case anyone still sees the issues described here feel free to reopen it.


import fails with panic

importing a tiff file that converts using tiff2pdf to a valid pdf (though does give warnings about the tiff file: TIFFReadDirectory: Warning, Unknown field with tag 33000 (0x80e8) encountered.
With pdfcpu import it fails with this:
pdfcpu import -vv  11.pdf 11.tiff
importing images into 11.pdf: [11.tiff]
Import config: A4 595x842 points, pos=full, dx=0, dy=0, scaling: 0.5 relative
11.pdf will be created
Fatal: unexpected panic attack: runtime error: index out of range
github.com/hhrutter/pdfcpu/pkg/api.Process.func1
/home/dhf/Development/go/src/github.com/hhrutter/pdfcpu/pkg/api/process.go:48
runtime.call32
/usr/local/go/src/runtime/asm_amd64.s:522
runtime.gopanic
/usr/local/go/src/runtime/panic.go:513
io/ioutil.readAll.func1
/usr/local/go/src/io/ioutil/ioutil.go:30
runtime.call32
/usr/local/go/src/runtime/asm_amd64.s:522
runtime.gopanic
/usr/local/go/src/runtime/panic.go:513
runtime.panicindex
/usr/local/go/src/runtime/panic.go:44
github.com/hhrutter/pdfcpu/ccitt.(*pixelBuf).addRun
/home/dhf/Development/go/src/github.com/hhrutter/pdfcpu/ccitt/reader.go:84
github.com/hhrutter/pdfcpu/ccitt.(*decoder).handleHorizontal
/home/dhf/Development/go/src/github.com/hhrutter/pdfcpu/ccitt/reader.go:348
github.com/hhrutter/pdfcpu/ccitt.(*decoder).handleHorizontal-fm
/home/dhf/Development/go/src/github.com/hhrutter/pdfcpu/ccitt/reader.go:578
github.com/hhrutter/pdfcpu/ccitt.(*decoder).decodeGroup4
/home/dhf/Development/go/src/github.com/hhrutter/pdfcpu/ccitt/reader.go:589
github.com/hhrutter/pdfcpu/ccitt.(*decoder).decode
/home/dhf/Development/go/src/github.com/hhrutter/pdfcpu/ccitt/reader.go:617
github.com/hhrutter/pdfcpu/ccitt.(*decoder).Read
/home/dhf/Development/go/src/github.com/hhrutter/pdfcpu/ccitt/reader.go:631
bytes.(*Buffer).ReadFrom
/usr/local/go/src/bytes/buffer.go:206
io/ioutil.readAll
/usr/local/go/src/io/ioutil/ioutil.go:36
io/ioutil.ReadAll
/usr/local/go/src/io/ioutil/ioutil.go:45
github.com/hhrutter/pdfcpu/tiff.Decode
/home/dhf/Development/go/src/github.com/hhrutter/pdfcpu/tiff/reader.go:683
github.com/hhrutter/pdfcpu/pkg/pdfcpu.ReadTIFFFile
/home/dhf/Development/go/src/github.com/hhrutter/pdfcpu/pkg/pdfcpu/imageRead.go:437
github.com/hhrutter/pdfcpu/pkg/pdfcpu.createImageResource
/home/dhf/Development/go/src/github.com/hhrutter/pdfcpu/pkg/pdfcpu/stamp.go:819
github.com/hhrutter/pdfcpu/pkg/pdfcpu.NewPageForImage
/home/dhf/Development/go/src/github.com/hhrutter/pdfcpu/pkg/pdfcpu/importImage.go:386
github.com/hhrutter/pdfcpu/pkg/api.ImportImages
/home/dhf/Development/go/src/github.com/hhrutter/pdfcpu/pkg/api/api.go:1347
github.com/hhrutter/pdfcpu/pkg/api.Process
/home/dhf/Development/go/src/github.com/hhrutter/pdfcpu/pkg/api/process.go:80
main.process
/home/dhf/Development/go/src/github.com/hhrutter/pdfcpu/cmd/pdfcpu/main.go:268
main.main
/home/dhf/Development/go/src/github.com/hhrutter/pdfcpu/cmd/pdfcpu/main.go:116
runtime.main
/usr/local/go/src/runtime/proc.go:201
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1333
Another file fails with this:
pdfcpu import -v  0.pdf 0.tiff
importing images into 0.pdf: [0.tiff]
Import config: A4 595x842 points, pos=full, dx=0, dy=0, scaling: 0.5 relative
0.pdf will be created
Fatal: ccitt: corrupt group4 data
Again this converts with tiff2pdf and I can merge them using pdfcpu merge creating a valid multipage pdf.

There are quite some TIFF extensions out there and TIFF per se is a versatile image format.
You have stumbled upon a bug in the CCITT Reader that may or may not have been caused by the nature of your TIFF file. What you are reporting makes sense to me because for merging image stream decoding is not necessary.
I am up to the challenge but I am afraid without a test.tiff that reproduces this I will have a hard time fixing this,
Cheers

Please send your contact information to me<EMAIL_ADDRESS>I can get you samples. We can discuss in private.
Get Outlook for iOShttps://aka.ms/o0ukef

From: Horst Rutter<EMAIL_ADDRESS>Sent: Monday, December 24, 2018 1:38 PM
To: hhrutter/pdfcpu
Cc: Donald French; Author
Subject: Re: [hhrutter/pdfcpu] import fails with panic (#58)
There are quite some TIFF extensions out there and TIFF per se is a versatile image format.
You have stumbled upon a bug in the CCITT Reader that may or may not have been caused by the nature of your TIFF file. What you are reporting makes sense to me because for merging image stream decoding is not necessary.
I am up to the challenge but I am afraid without a test.tiff that reproduces this I will have a hard time fixing this,
Cheers
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/hhrutter/pdfcpu/issues/58#issuecomment-449772778, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AABcq90U68aICyataPGx7fuHZnCh7onmks5u8UljgaJpZM4ZgnOq.

Fixed and part of the next release.


Incorrect translation of special characters

When attepmting to translate a citation from the following source:
@inproceedings{10.1145/2187980.2188259,
author = {Kaytoue, Mehdi and Silva, Arlei and Cerf, Lo\"{\i}c and Meira, Wagner and Ra\"{\i}ssi, Chedy},
title = {Watch Me Playing, i Am a Professional: A First Study on Video Game Live Streaming},
year = {2012},
isbn = {9781450312301},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/2187980.2188259},
doi = {10.1145/2187980.2188259},
abstract = {"Electronic-sport" (E-Sport) is now established as a new entertainment genre. More and more players enjoy streaming their games, which attract even more viewers. In fact, in a recent social study, casual players were found to prefer watching professional gamers rather than playing the game themselves. Within this context, advertising provides a significant source of revenue to the professional players, the casters (displaying other people's games) and the game streaming platforms. For this paper, we crawled, during more than 100 days, the most popular among such specialized platforms: Twitch.tv. Thanks to these gigabytes of data, we propose a first characterization of a new Web community, and we show, among other results, that the number of viewers of a streaming session evolves in a predictable way, that audience peaks of a game are explainable and that a Condorcet method can be used to sensibly rank the streamers by popularity. Last but not least, we hope that this paper will bring to light the study of E-Sport and its growing community. They indeed deserve the attention of industrial partners (for the large amount of money involved) and researchers (for interesting problems in social network dynamics, personalized recommendation, sentiment analysis, etc.).},
booktitle = {Proceedings of the 21st International Conference on World Wide Web},
pages = {1181–1188},
numpages = {8},
keywords = {e-sport, ranking, popularity prediction, starcraft ii, video game, twitch.tv, social community},
location = {Lyon, France},
series = {WWW '12 Companion}
}

This entry generates as follows:
Kaytoue, M., Silva, A., Cerf, L., Meira, W., & Ra"\issi, C. (2012). Watch Me Playing, i Am a Professional: A First Study on Video Game Live Streaming. In Proceedings of the 21st International Conference on World Wide Web (pp. 1181–1188). Association for Computing Machinery.
Ra\"\issi should be Raïssi

Thanks for your feedback. It seems that we'll need something like this but in Javascript.

Fixed in 1a38761


Blob Storage State Http Conditional Header Not Met Exception

Version
C# 4.0.8
Describe the bug
Every once in a while we get an exception traced back to the middleware where we call next. It looks like we are getting a 412 response back from Azure Blob Storage.
Expected behavior
No exception.
Additional context
Exception:
Exception caught : System.AggregateException: One or more errors occurred. (The condition specified using HTTP conditional header(s) is not met.) ---> Microsoft.WindowsAzure.Storage.StorageException: The condition specified using HTTP conditional header(s) is not met.
 at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token) in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\WindowsRuntime\Core\Executor\Executor.cs:line 316
 at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.DownloadRangeToStreamAsync(Stream target, Nullable`1 offset, Nullable`1 length, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext, IProgress`1 progressHandler, CancellationToken cancellationToken) in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\WindowsRuntime\Blob\CloudBlob.cs:line 438
 at Microsoft.WindowsAzure.Storage.Blob.BlobReadStream.DispatchReadAsync(Byte[] buffer, Int32 offset, Int32 count) in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\WindowsRuntime\Blob\BlobReadStream.cs:line 137
 --- End of inner exception stack trace ---
 at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
 at Microsoft.WindowsAzure.Storage.Blob.BlobReadStream.<>c__DisplayClass4_0.<Read>b__0() in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\WindowsRuntime\Blob\BlobReadStream.cs:line 80
 at Microsoft.WindowsAzure.Storage.Core.Util.CommonUtility.RunWithoutSynchronizationContext[T](Func`1 actionToRun) in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\Common\Core\Util\CommonUtility.cs:line 402
 at System.IO.StreamReader.ReadBuffer(Span`1 userBuffer, Boolean& readToUserBuffer)
 at System.IO.StreamReader.ReadSpan(Span`1 buffer)
 at System.IO.StreamReader.Read(Char[] buffer, Int32 index, Int32 count)
 at Newtonsoft.Json.JsonTextReader.ReadData(Boolean append, Int32 charsRequired)
 at Newtonsoft.Json.JsonTextReader.ParseValue()
 at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
 at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
 at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
 at Microsoft.Bot.Builder.Azure.AzureBlobStorage.<>c__DisplayClass7_0.<<ReadAsync>g__ReadIndividualKey|4>d.MoveNext() in D:\a\1\s\libraries\Microsoft.Bot.Builder.Azure\AzureBlobStorage.cs:line 135
 --- End of stack trace from previous location where exception was thrown ---
 at Microsoft.Bot.Builder.Azure.AzureBlobStorage.ReadAsync(String[] keys, CancellationToken cancellationToken) in D:\a\1\s\libraries\Microsoft.Bot.Builder.Azure\AzureBlobStorage.cs:line 115
 at Microsoft.Bot.Builder.BotState.LoadAsync(ITurnContext turnContext, Boolean force, CancellationToken cancellationToken) in D:\a\1\s\libraries\Microsoft.Bot.Builder\BotState.cs:line 67
 at Microsoft.Bot.Builder.BotState.BotStatePropertyAccessor`1.GetAsync(ITurnContext turnContext, Func`1 defaultValueFactory, CancellationToken cancellationToken) in D:\a\1\s\libraries\Microsoft.Bot.Builder\BotState.cs:line 284
 at Microsoft.Bot.Builder.Dialogs.DialogSet.CreateContextAsync(ITurnContext turnContext, CancellationToken cancellationToken) in D:\a\1\s\libraries\Microsoft.Bot.Builder.Dialogs\DialogSet.cs:line 64
 at EdinaRealty.Emma.Bot.v4.EmmaBot.OnTurnAsync(ITurnContext turnContext, CancellationToken cancellationToken) in F:\Development_New\EdinaRealty.Emma\EdinaRealty.Emma\EdinaRealty.Emma.Bot.v4\EmmaBot.cs:line 102
 at EdinaRealty.Emma.Bot.v4.Middleware.SpeakSettingMiddleware.OnTurnAsync(ITurnContext turnContext, NextDelegate next, CancellationToken cancellationToken) in F:\Development_New\EdinaRealty.Emma\EdinaRealty.Emma\EdinaRealty.Emma.Bot.v4\Middleware\SpeakSettingMiddleware.cs:line 42
 at Microsoft.Bot.Builder.MiddlewareSet.ReceiveActivityWithStatusAsync(ITurnContext turnContext, BotCallbackHandler callback, CancellationToken cancellationToken) in D:\a\1\s\libraries\Microsoft.Bot.Builder\MiddlewareSet.cs:line 55
 at Microsoft.Bot.Builder.BotAdapter.RunPipelineAsync(ITurnContext turnContext, BotCallbackHandler callback, CancellationToken cancellationToken) in D:\a\1\s\libraries\Microsoft.Bot.Builder\BotAdapter.cs:line 167
 ---> (Inner Exception #0) Microsoft.WindowsAzure.Storage.StorageException: The condition specified using HTTP conditional header(s) is not met.
 at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token) in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\WindowsRuntime\Core\Executor\Executor.cs:line 316
 at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.DownloadRangeToStreamAsync(Stream target, Nullable`1 offset, Nullable`1 length, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext, IProgress`1 progressHandler, CancellationToken cancellationToken) in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\WindowsRuntime\Blob\CloudBlob.cs:line 438
 at Microsoft.WindowsAzure.Storage.Blob.BlobReadStream.DispatchReadAsync(Byte[] buffer, Int32 offset, Int32 count) in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\WindowsRuntime\Blob\BlobReadStream.cs:line 137
 Request Information
 RequestID:d0b2a7f0-801e-000e-26a2-ae3bd9000000
 RequestDate:Thu, 17 Jan 2019 14:22:07 GMT
 StatusMessage:The condition specified using HTTP conditional header(s) is not met.
 ErrorCode:ConditionNotMet
 ErrorMessage:The condition specified using HTTP conditional header(s) is not met. RequestId:d0b2a7f0-801e-000e-26a2-ae3bd9000000 Time:2019-01-17T20:22:07.1367715Z
 <---

[bug]

Drilling into this a little I don't think this has anything to do with the IStoreItem ETag == "*" behavior the AzureBlobStorage code builds in. Firstly that code seems to basically work, secondly the ConversationState doesn't use that anyhow (because it saves a dictionary NOT an IStoreItem) and third this exception occurs on the read path anyhow!
However, looking closely at this code and it appears to have a couple of optimizations both of which would cause the Blob Storage Client to be used from multiple threads - this is absolutely wrong. That client is not thread safe.
In particular the read creates a collection of Tasks and then waits on them. This is wrong, each task would need its own blob storage client.
And then overall the class attempts to cache a _container again this is wrong as this class is often used as a singleton and so again could potentially have multiple threads.
So the fix here is to remove these optimizations.

Root Cause
The 412 status during a blob read happens if the blob is modified while being downloaded which can happen if two threads/servers are processing a turn for the conversation at the same time with one reading and the other writing. As such, this is just an Azure Storage specific manifestation of a concurrency issue.
A Simple Repro
Here's an app that took about 20mins to whip together that succintly repros the issue using nothing but the AzureBotStorage APIs. The project references the released 4.2.2 version of the package. However, if you manually reference the newer implementation with the changes made by the PR you will still observe the same issue.
AccessCondition not the problem
There was, AFAICT, nothing wrong with the AccessCondition logic. The "fix" provided by the PR simply flips the ternary condition around, resulting in no actual difference in behavior:

Regressions caused by the PR

The changes have reverted all the concurrent reading/writing around individual storage keys, which is detrimental to throughput because it serializes work that could otherwise be done in parallel.

The changes have introduced an additional HTTP request to Azure Storage for every call to the AzureBlobStorage::WriteAsync API in the form of constantly calling CreateIfNotExistsAsync to ensure the container exists. So basically, every on turn that saves state you will be making a minimum of two calls instead of one while the state is being persisted. This not only costs developers runtime performance in terms of network latency, but will cost actual 💰 in terms calls to Azure Storage.

Azure Storage SDK concurrency concerns
Despite the standard, boilerplate notes about thread safety on  the CloudBlobClient and related API docs, in practice they are known to be completely thread safe.
I have built and shipped some highly scalable, highly concurrent Azure Storage based solutions utilizing them in this way (for several years) and have never had an issue with the SDK in this regard. You can peruse the source code and see for yourself that the implementation is thread safe. Also, if there were issues at the SDK level with how AzureBlobStorage is using the APIs, I think those would have been surfaced when Southworks performed stress testing on the implementation or even now in my repro app.
And, yes, I am fully aware I am contradicting myself from 6yrs ago, but the SDK has changed drastically since then. 😆
There is no "real" fix for this
There is no automatic, one size all fix for concurrency issues. From an SDK perspectice, a good solution for this kind of problem would be to catch the 412 storage exception and translate that to an abstract exception that is married to the IStorage abstraction layer (e.g. an StorageConcurrencyException) with the  AzureBlobStorage doing that translation to when it encounters the 412. Likewise there will be concurrency issues with any other storage layer, such as CosmosDbStorage which would watch for its specific representation of this error and translate it to the same abstract exception.
This exception would next bubble up from the storage abstraction through the bot state abstraction in BotState::LoadAsync where, technically, it should be translated to another exception specific to that level of abstraction (e.g. an BotStateConcurrencyException). Next, that would bubble up through the IStatePropertyAccessor<T>::GetAsync and, technically, another abstraction would be good here too since the whole point of this interface is to decouple from the BotState implementation. Finally it would reach the bot/app layer where it could be caught and acted upon however the developer sees fit such as maybe retrying the call to GetAsync.
Final recommendations

I would highly recommend reverting the changes made in this PR based on the aforementioned regressions it has introduced when absolutely nothing was/is provably wrong with the existing implementation.
This issue regarding the 412 status should be reopened until it is properly addressed.

in 4.3 the blob storage provider is much improved and this issue (on the read) is addressed with retry.


[X11] Memory not being cleaned up?

Can someone explain why winit appears to be leaving behind some memory per window?
See https://github.com/tomaka/glutin/issues/1073:

The main cause of this issue is going to be however: https://i.imgur.com/1xhOKQl.png
Produced with:
extern crate glutin;

use std::thread;
use std::time::Duration;

struct Win {
    win: glutin::Window,
    ev: glutin::EventsLoop,
}

impl Win {
    fn new() -> Win {
        let ev = glutin::EventsLoop::new();
        let win = glutin::WindowBuilder::new()
            .with_dimensions(glutin::dpi::LogicalSize::new(300.0, 200.0))
            .build(&ev)
            .unwrap();

        Win { win, ev }
    }

    fn event(&mut self) -> (bool, bool) {
        let mut exit = false;
        let mut add = false;
        self.ev.poll_events(|e| match e {
            glutin::Event::WindowEvent { event, .. } => match event {
                glutin::WindowEvent::CloseRequested => {
                    exit = true;
                }
                glutin::WindowEvent::MouseInput { state, .. } => {
                    if state == glutin::ElementState::Released {
                        add = true;
                    }
                }
                _ => (),
            },
            _ => (),
        });

        (exit, add)
    }
}

fn main() {
    let mut wv = vec![];

    wv.push(Win::new());

    loop {
        let mut i = 0;
        while i < wv.len() {
            let (close, add) = wv[i].event();
            if close {
                wv = vec![]
            } else {
                i += 1;
            }
            if add {
                wv.push(Win::new());
                wv.push(Win::new());
                wv.push(Win::new());
                wv.push(Win::new());
                wv.push(Win::new());
                wv.push(Win::new());
                wv.push(Win::new());
                wv.push(Win::new());
                wv.push(Win::new());
                wv.push(Win::new());
                wv.push(Win::new());
            }
        }

        if wv.len() == 0 {
            // wv.push(Win::new());
            break;
        }
        thread::sleep(Duration::from_millis(1000));
    }
}

Investigated in the glutin issue I linked. Is a non issue.