[FLOC-4060] enable boto logging for aws provisioning module

This is recovered from the old code in flocker.agents.node.ebs.
The boto logging in the provisioning code has been broken since
the ebs code switched to boto3.

Thanks @avg-I
It looks like a good idea and I can imaging how it'll help with debugging.

This is recovered from the old code in flocker.agents.node.ebs.

Instead of duplicating the code in flocker.agents.node.ebs I think we should definitely take the opportunity to move the EliotLogHandler and _enable_boto_logging to flocker.common._logging (or something)

The boto logging in the provisioning code has been broken since the ebs code switched to boto3.

I poked around and found:

http://stackoverflow.com/a/30249685

We need to add the handler to the botocore logger rather than boto3.
I suggest we fix that here too.
Please answer or address the points above and resubmit for a final review.

I wanted this to be a quick, small change within the scope of the parent issue which  FLOC-3976, so that I could see counts and types of AWS requests.
However, I do agree that it makes sense to converge the boto logging code and put it into a common place.
Do you think that I should do that in the scope of this issue or if I should create a new specific issue for that?

Thanks @avg-I
Understood. If it helps you make progress then I agree, it's not worth getting bogged down with this.
Just merge when you get green builds and create follow up issues to re-use / refactor / fix  the boto / boto3 logging in ebs driver.

Also looks like the branch needs merging forward.

Thank you, Richard! I've created FLOC-4141 for converging the boto logging.


Class hierarchy not shown on the website for Brick v1.1

Hi!
As of now, I am not able to see the class hierarchy of the ontology for Brick v1.1. I do so for v1.2 and v1.3 though.

I think I may have fixed the issue in this PR: https://github.com/BrickSchema/brick-website/pull/159
I'll see if the build is fixed and then I'll update the website today or tomorrow
(As an aside -- you should be using more recent versions of Brick! 1.3 is released now and is much improved over 1.1. Let me know if you have any issues with upgrading)

Should be fixed now!


Content model: address image alt text

With the first pass of content model work, we have not documented image alt text.
Task
Beside Image component itself, we need to revisit all components with images, and find out if the image inside need alt text. We are currently unsure if say an image inside Card would need alt text.

Updated in content model master spreadsheet
https://ibm.box.com/s/2drh0lhqrd8eo5tfc8ru46kdp8ckuu1c


Server can't process early data if connection failed to negotiate TLS 1.3

if in the connection client advertised version of protocol that is not supported by server (at 1a50eedf2a1fbb1e that means 0x0304 (TLS1.3 final) or 0x7f1d (127, 29; draft-29)) but includes 0-RTT, the connection will negotiate TLS 1.2 and server will fail to handle the early data correctly (i.e. it won't continue the connection)
see also #6734 for a related, but different issue
Note: the test case uses 0x0305 as the test version (to not require changes after TLS 1.3 goes final), behaviour can be changed by editing line 605
reproducer
openssl req -x509 -newkey rsa -keyout localhost.key -out localhost.crt -subj /CN=localhost -nodes -batch
openssl s_server -key localhost.key -cert localhost.crt -www 2>server.err >server.out &
openssl_pid=$!
git clone https://github.com/tomato42/tlsfuzzer
pushd tlsfuzzer
# won't be needed after https://github.com/tomato42/tlsfuzzer/pull/423 is merged
git checkout 0rtt-garbage-resumption
git clone https://github.com/tomato42/tlslite-ng .tlslite-ng
ln -s .tlslite-ng/tlslite tlslite
git clone https://github.com/warner/python-ecdsa .python-ecdsa
ln -s .python-ecdsa/ecdsa ecdsa
PYTHONPATH=. python scripts/test-tls13-0rtt-garbage.py
popd
kill $openssl_pid

tlsfuzzer output
note the "handshake with invalid 0-RTT and unknown version (downgrade to TLS 1.2)" test case
failure in "handshake with 0-RTT, HRR and early data after 2nd Client Hello" is caused by #6734
sanity ...
OK

handshake with invalid 0-RTT and CCS ...
OK

handshake with invalid 0-RTT and HRR ...
OK

handshake with invalid 0-RTT with fragmented early data ...
OK

handshake with invalid 0-RTT and unknown version (downgrade to TLS 1.2) ...
Error encountered while processing node <tlsfuzzer.expect.ExpectChangeCipherSpec object at 0x7f20988a18d0> (child: <tlsfuzzer.expect.ExpectFinished object at 0x7f20988a1910>) with last message being: <tlslite.messages.Message object at 0x7f20988a6c10>
Error while processing
Traceback (most recent call last):
  File "scripts/test-tls13-0rtt-garbage.py", line 727, in main
    runner.run()
  File "/home/hkario/dev/tlsfuzzer/tlsfuzzer/runner.py", line 210, in run
    RecordHeader2)))
AssertionError: Unexpected message from peer: Alert(fatal, unexpected_message)

undecryptable record later in handshake together with early_data ...
OK

handshake with invalid 0-RTT and CCS between early data records ...
OK

handshake with invalid PSK ...
OK

handshake with 0-RTT, HRR and early data after 2nd Client Hello ...
Error encountered while processing node <tlsfuzzer.expect.ExpectNewSessionTicket object at 0x7f209888a750> (child: <tlsfuzzer.expect.ExpectNewSessionTicket object at 0x7f209888a750>) with last message being: None
Error while processing
Traceback (most recent call last):
  File "scripts/test-tls13-0rtt-garbage.py", line 727, in main
    runner.run()
  File "/home/hkario/dev/tlsfuzzer/tlsfuzzer/runner.py", line 196, in run
    raise AssertionError("Unexpected closure from peer")
AssertionError: Unexpected closure from peer

handshake with invalid 0-RTT ...
OK

sanity ...
OK

Basic check if TLS 1.3 server can handle 0-RTT handshake
Verify that the server can handle a 0-RTT handshake from client
even if (or rather, especially if) it doesn't support 0-RTT.

version: 1

Test end
successful: 9
failed: 2
  'handshake with 0-RTT, HRR and early data after 2nd Client Hello'
  'handshake with invalid 0-RTT and unknown version (downgrade to TLS 1.2)'

openssl output
Using default temp DH parameters
ACCEPT
140464152045376:error:140940F5:SSL routines:ssl3_read_bytes:unexpected record:ssl/record/rec_layer_s3.c:1724:
140464152045376:error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac:ssl/record/ssl3_record.c:674:

If it negotiates TLS 1.2, I'm not sure what you think is wrong.

the problem is not the negotiation of TLS 1.2, the problem is that if negotiation of TLS 1.2 is a result of mismatch between supported versions (supported draft) of TLS 1.3, the server will abort the connection because it doesn't recognise the early data for what it is
section D.3 is a bit confusing though, speaking about only the client, not server
and together with the downgrade protection from section 4.1.3 a client can know if it connected to a TLS 1.3 server or not, even if the connection ended up with TLS 1.2

I'm still not sure what you're trying to say. If there is early
data on a TLS 1.2 connection, because that is what it ends up
negiotating, shouldn't it complain because it has no idea what
this early data is, and abort?

Right, there's not much point since existing TLS 1.2 servers will not do this. So, when we think about operation concerns with a large server deployment, the incompatibility is already an issue. Note the client text has a MUST-level requirement to reject 1.2 ServerHellos that makes this pointless anyway. (The requirement is needed to make the robust version of the client retry possible.)

section D.3 is a bit confusing though, speaking about only the client, not server

No, D.3 talks about both clients and servers. Here is the server text.

To avoid this error condition, multi-server deployments SHOULD ensure
a uniform and stable deployment of TLS 1.3 without 0-RTT prior to
enabling 0-RTT.

But I agree the text isn't ideal. My original version in https://github.com/tlswg/tls13-spec/pull/439 more explicitly said to do a retry. Some of that got lost in https://github.com/tlswg/tls13-spec/pull/502, which I +1'd but, looking again, I think it got too weak.
As far as I'm concerned, the application-level client retry on unexpected TLS 1.2 ServerHello is a requirement for playing the 0-RTT game. That was a pretty common sentiment from server operators, that this fallback is needed. While everyone SHOULD get the deployment order right, it's a bit of an operational concern.
Alas, It's an application-level retry, so it's difficult to mandate this at TLS, but so it goes. With a good 0-RTT API from the TLS layer and the right application-level organization, it turns out you basically get the client retry for free in your 0-RTT reject code.

I agree with the above. This seems like correct behaviour to me?

I agree with the above. This seems like correct behaviour to me?

yes, it would be the correct behaviour, though introducing the retry mechanism I'm really not a fan of and in a place that it's not really necessary

Alas, It's an application-level retry, so it's difficult to mandate this at TLS, but so it goes. With a good 0-RTT API from the TLS layer and the right application-level organization, it turns out you basically get the client retry for free in your 0-RTT reject code.

so should we verify that the s_client is performing that 0-RTT retry in case the TLS 1.3 was rejected?
(I haven't looked at the openssl test suite, so it may already be there, if it is, then this bug can be closed)

so should we verify that the s_client is performing that 0-RTT retry in case the TLS 1.3 was rejected?

It doesn't. But I'm not really sure it's appropriate for s_client to do so. It might make sense for browsers and other applications - but perhaps not for s_client as a testing tool.

I see s_client both as a testing tool and example code
if that code wouldn't be in s_client, where should it be?

I see s_client both as a testing tool and example code

Yikes...don't use s_client as an example :-)
Perhaps there should be something in the man page, or on the wiki?

Yikes...don't use s_client as an example :-)

not for copy-pasting, more like "how do I use this API method"


Injecting Apple Music disconnects it from network

Title

beta4.2?

Yes

It seems like it only disconnects it from Wi-Fi because when I open data it works again.

@roothide @dleovl

what is your device region? as far as we know, devices in some regions will restrict the app’s networking permissions.

It’s Greek. And how does it make sense it only restricts wifi access especially, only when it’s injected💀

can you try reinstalling apple musci app?

Already tried it

I've got this too. really irritating

Seems to be fixed as of latest release. Make a new issue if the error persists.


Fixes issue: #1812 Domain: C/CPP

Added documentation for Macros in C.

Description 📜
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context.
Fixes #2227
Added documentation for Macros in C.
Fixes the changes requested in PR 2227.

Type of change 📝

[x] Documentation (Content Creation in the form of codes or tutorials)

Domain of Contribution 📊

[x] C/CPP

Checklist ✅

[x] I follow Contributing Guidelines & Code of conduct of this project.
[x] I have performed a self-review of my own code or work.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] My changes generates no new warnings.
[x] I'm GWOC'21 contributor

Screenshots / Gif (Optional) 📸

@Saviour1001 please check if the changes requested have been made and all's well and good on this PR. Thanks!

@vasu-1 please check if the changes made are sufficient. Thanks!

Niceee @KartikChinda . Once Merging is done kindly close the issue of the respective Pr.
Good to go for merge @Saviour1001

Sure thing!


http: reduce delay-close issues for HTTP/1.1 and below

skipping delay close for HTTP/1.0 framed by connection close (as it simply reduces time to end-framing) as well as HTTP/1.1 iff the request is fully read (so there's no FIN-RST race)
Risk Level: low
Testing: new unit tests
Docs Changes: n/a
Release Notes: inline
Addresses the Envoy-specific parts of https://github.com/envoyproxy/envoy/issues/19821
Runtime guard: envoy.reloadable_features.skip_delay_close

Not really a productive comment but this was super helpful. Thank you!


Sentry

What changes are being made and why?
Added Sentry alert plugin. Which sends event to sentry on execution fails
How the changes have been QAed?

Was created test project in Sentry. Through unit test was added as url sentry URL, and provided body into sentry.yaml flow
id: sentry
namespace: io.kestra.tests

inputs:
  - name: url
    type: STRING

listeners:
  - tasks:
      - id: sentry
        type: io.kestra.plugin.notifications.sentry.SentryExecution
        url: "https://{URI}/api/{PROJECT_ID}/store/?sentry_varsion=7&sentry_clien=java&sentry_key={PUBLIC_KEY}"
        level: "error"
        serverName: "{{ inputs.url }}"
        extra:
          title: "Kestra Sentry alert"
          service: "Failed on task `failed`"
          url: "{{ inputs.url }}"

tasks:
  - id: seq
    type: io.kestra.core.tasks.flows.Sequential
    tasks:
    - id: failed
      type: io.kestra.core.tasks.executions.Fail

Setup Instructions

DSN contains all required information to send events to sentry
Event payload Information about event payload

Wrong message in commit. Commits are related to sentry only (Not zenduty)

closes https://github.com/kestra-io/plugin-notifications/issues/89
Suggestions:

remove commas in the example flow
Execution task returns an empty body for me using the following flow -- could you validate?

id: sentry
namespace: dev

inputs:
  - name: url
    type: STRING
    defaults: https://xxxxxxx.ingest.sentry.io/api/xxxxx/security/?sentry_key=xxxxxx
      #<EMAIL_ADDRESS>
listeners:
  - tasks:
      - id: sentry
        type: io.kestra.plugin.notifications.sentry.SentryExecution
        url: "{{ inputs.url }}"
        level: "error"
        serverName: "{{ inputs.url }}"
        extra:
          title: "Kestra Sentry alert"
          service: "Failed on task `failed`"
          url: "{{ inputs.url }}"

tasks:
  - id: failed
    type: io.kestra.core.tasks.executions.Fail

@iNikitaGricenko could you maybe document more step by step how do you expect this to be used?
You shared a URL in the flow from the PR body:
url: "https://{URI}/api/{PROJECT_ID}/store/?sentry_varsion=7&sentry_clien=java&sentry_key={PUBLIC_KEY}"

here, you don't use the URL from inputs, did you just forgot?
also, the URL structure shown here doesn't match the DSN in Sentry:

the task documentation is even more confusing as it additionally splits URI into host and URI, and adds serverName property pointing to localhost:
format: https://{HOST/URI}/api/{PROJECT_ID}/store/?sentry_version=7&sentry_client=java&sentry_key={PUBLIC_KEY}

my understanding is that Sentry cannot be self-hosted. in that case, localhost doesn't make sense. I could be missing something and if so, can you help me understand it?

Currently, I'm getting a 404 error by following your instructions to use the project's DSN as URL

Sorry, you right, my fault. DSN is just url that should be parsed to parts, but never used to send requests. Will fix that. Might be will be easier to make users provide DSN as url and then just parse it to api request

@anna-geller Can you try if this changes make it easier?

@anna-geller You saying that plugin wont work you not provide full payload? It's weird because actually i'm sendy without and with payload and it's works okay. Might be something wrong with event_id? As i see you set it to execution id with is wrong, because event_id expects uuid4 format, but might be i'm wrong. Will check that

you're right; without including event_id, it works
but when skipping payload entirely, I'm getting this on Alert task:

This is minimal requirements (which is only dsn):

So can you provide your yaml that fails?

I said Alert task, not Execution task :)
This one won't work without payload even though payload is marked as optional:
id: sentry_alert
namespace: dev

tasks:
  - id: fail
    type: io.kestra.core.tasks.executions.Fail

errors:
  - id: alert_on_failure
    type: io.kestra.plugin.notifications.sentry.SentryAlert
    dsn<EMAIL_ADDRESS>


[4.x] Add xml declaration to the xml error renderer

This PR addresses issue #2747.

Coverage remained the same at 100.0% when pulling 9e9f23c849b3c376bd6a376d79d7c03bdbbee9b9 on adriansuter:patch-xmlErrorRendererDeclaration into 05456641fc90be5cae3e5d950e6ec2c6d5df7e6c on slimphp:4.x.


[Bug]: Incorrect color for the Secondary Ghost button text in Workleap's mode.

Contact Details
<EMAIL_ADDRESS>What happened?
The button component has incorrect text color in the default state for secondary/ghost appearance in the Workleap's mode. The correct token should be --hop-neutral-text-weak (#6C6C6B), but at the moment it is using --ids-btn-text-ghost-secondary which resolves into --hop-neutral-text (#3c3c3c).

Result

Design

Actual

Component
Button
Component Version
0.9.1
Which browsers are you seeing the problem on?
Chrome
Mobile Device
No response
Relevant log output
No response

Thanks for the report, we had a card for this, this will be tackled soon.

This has been fixed


Facilitate debug

Remove unnecessary attributes of debuged task algorithm tab. - Leave only debug url and name.
Add copy of debug url inline in the graph from the debug node.

Add the url to the graph as well, and allow copy


add method to get list of timezones and bump Android dependencies

Was able to get the plugin working in my IDE again so here's a PR for #8
Updated the plugin to use AndroidX and updated some .gitignore rules as well

@pinkfish noticed this has been merged for a while but not released. Any plans to publish an updated version of the plugin to pub.dev? Currently I've got the example app for the flutter_local_notifications retrieving the timezone via platform channels. If you're still maintaining this plugin then having the community use an existing plugin (yours) would be easier though flutter_local_notifications also supports macOS so would need this https://github.com/pinkfish/flutter_native_timezone/pull/18 merged in and released as well

@pinkfish noticed this has been merged for a while but not released. Any plans to publish an updated version of the plugin to pub.dev? Currently I've got the example app for the flutter_local_notifications retrieving the timezone via platform channels. If you're still maintaining this plugin then having the community use an existing plugin (yours) would be easier though flutter_local_notifications also supports macOS so would need this https://github.com/pinkfish/flutter_native_timezone/pull/18 merged in and released as well

@pinkfish As I haven't heard anything for a while and how it looks like there's open issues/PRs with no updates/responses, it looks as though the plugin isn't being maintained that I'm likely going to look at publishing a separate plugin. Wanted to drop a courtesy message on this to give you a heads up in case you do plan to go back to maintaining the plugin and because you had made contributions to flutter_local_notifications before


feat(tui): highlight search results

Description of change
closes #2
Thanks @pythops for the pointer!
How has this been tested? (if applicable)

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
Attention: Patch coverage is 0% with 24 lines in your changes missing coverage. Please review.

Project coverage is 7.71%. Comparing base (2d727cd) to head (94c88ea).

Files
Patch %
Lines

src/tui.rs
0.00%
24 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff            @@
##            main     #61      +/-   ##
========================================
- Coverage   7.83%   7.71%   -0.13%
========================================
  Files         13      13
  Lines        740     752      +12
========================================
  Hits          58      58
- Misses       682     694      +12

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


webpack in deps

Hi, Thanks for this lib. Any specific reason for the webpack in the direct dependencies?

It generates a lot of deprecation warnings and the tests pass when it is removed from the dependencies.

I think it should  be a development dependences. It is used for testing the use in browsers.

Moved it to development dependencies.


Enable useUnifiedTopology in the Mongoose adapter

Addresses this warning:
- Connecting to database(node:16844) DeprecationWarning: current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUnifiedTopology: true } to the MongoClient constructor.

See also: https://github.com/mongodb/node-mongodb-native/releases/tag/v3.2.1

Did I do that right?


fix(iOS): Ensure reports on disk prior to app termination

Goal
(Extends #287)
Ensure report delivery while holding the native JS execution thread for as little time as possible in the case of non-fatal errors being reported. Updated the integration test to make the stack trace ridiculous.
Tests

Tested against physical devices as well as in the integration suite

Linked issues
Closes #287

[ ] Internal consistency with the rest of the library - is there any overlap between existing interfaces and any which have been added?
[ ] Usage friction - is the proposed change in usage cumbersome or complicated?
[ ] Performance and complexity - are there any cases of unexpected O(n^3) when iterating, recursing, flat mapping, etc?
[ ] Concurrency concerns - if components are accessed asynchronously, what issues will arise
[ ] Thoroughness of added tests and any missing edge cases
[ ] Idiomatic use of the language

Both nice catches, thanks @fractalwrench !


client-side navigation doesn't work if endpoint has the same name as a route component

Describe the bug
The client-side navigation ~sveltekit:prefetch directive~ appears to not ~fire the prefetch~ work if there is an endpoint with the exact name of the page
Reproduction

Initialize template
Add route test.svelte
Add endpoint test.ts with a post function
Add /test to the Header component with sveltekit:prefetch, as the others

When hovering the "about" page, prefetch is fired, but when hovering the "test" rout it is not, and when clicking the link the site will do a full page reload to navigate there
Logs
No response
System Info
System:
    OS: Linux 5.4 Linux Mint 20.2 (Uma)
    CPU: (2) x64 Pentium(R) Dual-Core  CPU      E5400  @ 2.70GHz
    Memory: 319.32 MB / 3.84 GB
    Container: Yes
    Shell: 5.0.17 - /bin/bash
  Binaries:
    Node: 14.17.5 - /usr/bin/node
    npm: 6.14.14 - /usr/bin/npm
  Browsers:
    Chrome: 92.0.4515.159
    Firefox: 90.0
  npmPackages:
    @sveltejs/kit: next => 1.0.0-next.153
    svelte: ^3.34.0 => 3.42.1

Severity
serious, but I can work around it
Additional Information
No response

I suppose that is the intended behavior. If you see the documentation for Fall through routes, it clearly says the endpoints are tried first for a match. Since we find a match with the endpoint prefetch is not performed.

@anudeepreddy But the endpoint does not export a GET route, just a POST one

@GCastilho, Currently we obtain an array of routes then perform a check if it's an incredible endpoint. The current logic doesn't check what http methods are exposed in that route.

Ok, got it. But is there any plans to change this or it's considered the correct behavior?

I've just noticed that it's not just prefetch that isn't working, it's the whole client-side navigation that is
The site does a full page reload if there is an endpoint with the same name as a route component even if there are no get methods on the endpoint

I seem to be running into this too. Not sure but this I'm observing the following: Going to from route A to B works, but going back to route B doesn't update the view. The page endpoint is called consistently, and  it does seem to update the data, but only the root fields, deeper nested fields don't update.
So if the data has the following type:
type data  = {
  id: string,
  something: {
    nested: string[]
  }
}

then data.id does update on very client side navigation, but data.something.nested[0] doesn't.
Adding the following "works" but seems to be a hack to me:
<script lang="ts" context="module">
    export const router = false;
</script>

It seems a production build doesn't update the view the first time either, so going from A to B never renders B.

I found out if I flatten the data coming from the page endpoint I can get around the issue.
So from the original data:
type data  = {
  id: string,
  something: {
    nested: string[]
  }
}

i needed: data.something.nested[0]. If I return this value and thus the following type from the endpoint the component updates as I would expect:
type data  = {
  id: string,
  something:string // which is data.something.nested[0] mapped in the endpoint
}

Also returning the array directly works as well:
type data  = {
  id: string,
  something:string[] // which is data.something.nested mapped in the endpoint
}

Sorry for "spamming" this issue. I'm very new to Svelte / SvelteKit so probably this is just my misunderstanding Svelte components and maybe has not much to do with this issue, but maybe it's useful to others reading this.

I can no longer reproduce the original issue. See this demo on StackBlitz -- the /test route is prefetched and uses client-side navigation. Since this issue was filed, things are notably different, since an endpoint with the same name as a route component is treated as a page endpoint.
@yn5 your issue sounds different from the original issue. Can you open a separate issue, preferably with a full reproduction on StackBlitz?


Add existing encodings/ciphers from Ciphey

This is a list of all the currently supported encodings/ciphers in Ciphey.
We need to make them in Rust for Ares to achieve feature parity.
Encodings

[x] Base2 (Binary)
[ ] Base8 (Octal)
[ ] Base10 (Decimal)
[x] Base16 (Hexadecimal)
[x] Base32
[x] Base58 Bitcoin
[x] Base58 Flickr
[x] Base58 Ripple
[x] Base58 Monero (Not in Ciphey)
[ ] Base62 #137
[x] Base64
[x] Base64 URL
[ ] Base69
[ ] Base85
[x] Citrix CTX1 (Not in Ciphey)
[ ] Z85
[ ] ASCII Base85
[x] Base91
[x] Base65536
[ ] ASCII
[x] Reversed text
[x] Morse Code
[ ] DNA codons
[x] Atbash
[ ] Standard Galactic Alphabet (aka Minecraft Enchanting Language)
[ ] Leetspeak
[ ] Baudot ITA2
[x] URL encoding
[ ] SMS Multi-tap
[ ] DMTF
[ ] A1Z26
[ ] Prisoner's Tap Code
[ ] UUencode
[ ] Braille (Grade 1)

Ciphers

[x] Caesar Cipher
[ ] ROT47 (up to ROT94 with the ROT47 alphabet)
[ ] ASCII shift (up to ROT127 with the full ASCII alphabet)
[ ] Vigenère Cipher
[ ] Affine Cipher
[x] Railfence Cipher (Not in Ciphey)
[ ] Binary Substitution Cipher (XY-Cipher)
[ ] Baconian Cipher (both variants)
[ ] Soundex
[ ] Transposition Cipher
[ ] Pig Latin

Modern day cryptography

[ ] Repeating-key XOR
[ ] Single XOR

Esoteric languages

[ ] Brainfuck

Compression Methods

[ ] GZip

binary bytes type base 62 quick reference.https://www.codeproject.com/articles/1076295/base-encode

binary bytes type base 62 quick reference.https://www.codeproject.com/articles/1076295/base-encode

There's a broken PR for that here :D https://github.com/bee-san/Ares/pull/137

ASCII85 and Base85 are the same encoding, just different names. Maybe one of them should be removed from the list to avoid confusion.

ASCII85 and Base85 are the same encoding, just different names.

That's false, they're different. ASCII85 refers to Adobe's variation of base85.
They're separate functions in Python.
https://en.wikipedia.org/wiki/Ascii85#Adobe_version

Adobe adopted the basic btoa encoding, but with slight changes, and gave it the name Ascii85.

https://docs.python.org/3/library/base64.html
base64.a85encode(b, *, foldspaces=False, wrapcol=0, pad=False, adobe=False)
Encode the bytes-like object b using Ascii85 and return the encoded bytes.

base64.b85encode(b, pad=False)
Encode the bytes-like object b using base85 (as used in e.g. git-style binary diffs) and return the encoded bytes.

https://stackoverflow.com/questions/34969947/what-is-the-difference-between-a85encode-and-b85encode

Oh, right, I forgot how many implementations of Base85 there are. Perhaps we should specify it's the Adobe implementation of Base85 up there, for extra clarity.

With ciphers that require a password, like Vigenère Cipher, how is that done? Is the password just brute-forced?

With ciphers that require a password, like Vigenère Cipher, how is that done? Is the password just brute-forced?

Yes, a Vigenere cracker needs to be implemented.
Here's an example of one: https://www.guballa.de/vigenere-solver
That's the best one that I know of.
How it works: https://www.guballa.de/bits-and-bytes/implementierung-des-vigenere-solvers

A lot of cipher require you to know the key. We should write a crate for brute-forcing keys, and use that among all the cipher crates.

For Vigenere and XOR, I found these two algorithms:

https://github.com/TheAlgorithms/Rust/blob/master/src/ciphers/vigenere.rs
https://github.com/TheAlgorithms/Rust/blob/master/src/ciphers/xor.rs

Not sure if it's worth a whole issue or not but eventually instead of guessing/brute forcing the password, I think there should be a flag to provide a known password, then ares could only go over ciphers with passwords using the provided password.


prog: introduce the readable struct attribute

A more global issue: https://github.com/google/syzkaller/issues/1070
Context
An email from Ted:

Show
C strings instead of hex digits? It will make the reproducer much
more human understandable, as well making it easier to edit the string
when the developer is trying to do a better job minimizing the test
case than syzbot. For example:

memcpy(
(void*)0x20000000,
"\x6e\x6f\x75\x73\x65\x72\x5f\x78\x61\x74\x74\x72\x2c\x61\x63\x6c\x2c\x64"
"\x65\x62\x75\x67\x5f\x77\x61\x6e\x74\x5f\x65\x78\x74\x72\x61\x5f\x69\x73"
"\x69\x7a\x65\x3d\x30\x78\x30\x30\x30\x30\x30\x30\x30\x30\x30\x30\x30\x30"
"\x30\x30\x38\x30\x2c\x6c\x61\x7a\x79\x74\x69\x6d\x65\x2c\x6e\x6f\x62\x68"
"\x2c\x71\x75\x6f\x74\x61\x2c\x00\x3d\x93\x09\x61\x36\x5d\x73\x58\x9c",
89);

Would be *much* more understable if it were:

memcpy(
(void*)0x20000000,
"nouser_xattr,acl,debug_want_extra_isize=0x0000000000000080,lazytime,nobh,quota,",
80);

In that particular case, it was apparently already squashed and therefore treated as a byte blob, but normally the printed operations are also far from being most concise. E.g.:
  memcpy((void*)0x20000040, "ext4\000", 5);
  memcpy((void*)0x20000000, "./bus\000", 6);
  memcpy((void*)0x200000c0, "usrjquota=", 10);
  *(uint8_t*)0x200000ca = 0x2c;
  memcpy((void*)0x200000cb, "abort", 5);
  *(uint8_t*)0x200000d0 = 0x2c;
  memcpy((void*)0x200000d1, "debug_want_extra_isize", 22);
  *(uint8_t*)0x200000e7 = 0x3d;
  sprintf((char*)0x200000e8, "0x%016llx", (long long)0x2c);
  *(uint8_t*)0x200000fa = 0x2c;
  memcpy((void*)0x200000fb, "test_dummy_encryption", 21);
  *(uint8_t*)0x20000110 = 0x2c;
  memcpy((void*)0x20000111, "nojournal_checksum", 18);
  *(uint8_t*)0x20000123 = 0x2c;
  memcpy((void*)0x20000124, "noauto_da_alloc", 15);
  *(uint8_t*)0x20000133 = 0x2c;
  *(uint8_t*)0x20000134 = 0;

Proposal
Currently we rely on isReadableDataType  when we determine whether to print a value as a byte array or as a string: https://github.com/google/syzkaller/blob/1dac8c7a01e2bdd35cb04eb4901ddb157291ac2d/prog/encoding.go#L951-L953
Add a new readable struct/union attribute to let users indicate that, when generating a C reproducer, this particular attribute should be squashed and printed as a human-readable string. E.g.
type fs_options[ELEMS] {
	elems	array[fs_opt_elem[ELEMS]]
	common	array[fs_opt_elem[fs_options_common]]
	null	const[0, int8]
} [packed, readable]

Questions
Are there any other cases where it can be helpful?
A related (but not exactly the same) scenario are network packets, where it might be better to copy raw packets as a whole, rather than many small memcpy operations.

There is plenty of such string-forming structs, lots can be found $ grep "'.'" sys/*/*.txt (+ some that were added before char literals).


Drop submodule support

Submodule support TBH was kind of rushed in without proper discussion or planning, being an unrelated change wrapped up in a PR to improve performance.
Right now submodule support is not great IMO.

Overrides can be confusing and obscure.
Most commonly users won't even see the submodule stats, as all submodules are vendored.
Detecting submodules as vendored is hacky, as its not organized with Vendored, but sits in the main implementation as its own boolean value is_submodule.
Submodules can't even be analyzed unless they are initialized, potentially confusing for users.
Because of the above, repositories with clean histories checked out at the same commit can have different results.

For these reasons, I think that submodules should be skipped. I'd rather drop submodule support completely and potentially add it back in with proper support, than try and patch in proper support to the current implementation.
If a user is determined to analyze submodules it's fairly easy to do without support.

As a library, discover each submodule and run Gengo::analyze on it.
As a binary, iterate over submodules (this should be trivial with git submodule foreach).

As a binary, iterate over submodules (this should be trivial with git submodule foreach).

There is support for that in gix as well.

There is support for that in gix as well.

Awesome! I'm still considering the best way to support submodules, bub knowing that gix supports this adds some possibilities:

Re-export one or more types from gix
Document suggestions for how one could handle submodules
A combination of the above

My feeling is that submodule handling should be an integrated part of gengo because it is information that should be leveraged when analysing work trees (i.e. the checkout of a commit on disk). Otherwise, while traversing files, one would traverse into submodules and not know about it.
These do have .git files typically  (or .git directories) so they can be excluded from traversal, even without explicitly querying them using gix.
Maybe that submodule-ignoring behaviour, by default, would also be in the interest of onefetch. Over time, it can be expended once it becomes clearer on how to handle and/or classify submodules.

it is information that should be leveraged when analysing work trees (i.e. the checkout of a commit on disk). Otherwise, while traversing files, one would traverse into submodules and not know about it.

I think you've touched on something important here (and also read my mind a bit :laughing:).
When analyzing a revision, I think gengo shouldn't analyze submodules because the usage can be obscure and confusing. However, if gengo analyzed the files on disk, it should treat a submodule just like any other directory and analyze it. It's not necessarily obvious that the submodules must be initialized (git submodule update --init) when reading from a git revision like HEAD. However, it's very obvious when reading from the disk -- if the files aren't in the work tree, they don't get analyzed when reading from disk.
So my idea is that submodule support should be wrapped up with work tree support. In fact, it would be confusing if submodules weren't supported when reading files from the disk. :laughing: It's one of those scenarios where it will be harder to not do it than to do it.
This does raise thoughts and concerns that should probably go in an "Add support for analyzing worktrees" issue. But to document them while they're on my mind

Like tokei, worktrees should be generic, not require a git repository
Read from .gitignore or another ignore file to know which files/dirs to exclude? I think not, because, unlike tokei, we mark files as generated, vendored, etc. That does mean we read files that might be excluded, but opens up granular behavior like --include-generated.
How should overrides be handled? With the git implementation we use .gitattributes, but for a worktree I don't think we should. Too git-specific for a generic worktree implementation. There will probably be a lot of back-and-forth on this, so I'll save my ideas on this for another issue.
As far as the binary goes, this should perhaps be divided into subcommands gengo git and gengo worktree (and git-gengo can be an alias for gengo git).

Renaming issue to be more of an action, and closing as this action is completed.

As I am not sure if it came across, let me state my primary concern: If git is available, even when reading the work tree directly, I think gengo should use the available .gitattribute information, instead of ignoring it. However, it should treat this as optional.
The conclusions presented sound a lot like ignoring .gitattributes even if present, which wouldn't be what I expect as a user that went through the trouble of adding gengo specific attributes.

As far as the binary goes, this should perhaps be divided into subcommands gengo git and gengo worktree (and git-gengo can be an alias for gengo git).

If gengo is a developer tool (mainly) for debugging the library and running it in the real-world, maybe that's the right choice. But for maximum convenience, it's my hunch that one can make it so that 'it does the right thing' while allowing users to be specific about what they want. For instance, gengo HEAD or gengo @~1 is clearly a revspec, but just gengo . is clearly something related to the current directory. Not all options are unambiguous here, but I think that could be dealt with decently as well. Of course, this isn't a fully fleshed out description of possible UX, it's merely a description of my 'feeling' for how I would want to use it.

As I am not sure if it came across, let me state my primary concern: If git is available, even when reading the work tree directly, I think gengo should use the available .gitattribute information, instead of ignoring it. However, it should treat this as optional.

First, keep in mind that any overrides should be the exception, not the rule. GitHub has about 40k .gitattributes with linguist-* attributes, and 132k .gitattributes without any linguist-* attributes. On top of that, the vast majority probably don't have any .gitattributes at all. So I don't think there will be a lot of wasted hard work if in some circumstances we don't read from .gitattributes.
I get that it might be confusing if the git implementation uses git attributes, while a generic implementation doesn't. However, I believe a generic implementation should be fully generic, not a "maybe-git" implementation. If the generic implementation reads gitattributes, then I believe this can lead to some awkwardness because we're then left with these choices:

Ask users to put .gitattributes in non-git directories (Mercurial, SVN, or a plain old directory)

If we do this we'd also have to search up directories to find .gitattributes, with possibly no clear stopping point until /

Implement multiple ways to override (.gitattributes + config file) which leads to code complexity and questions about which should take precedence

Also, IMO setting overrides in .gitattributes is, TBH, a bit obscure. It's used for reading from a rev, and IMO the best way to do it since we can support reading from bare repos (thanks again for getting .gitattributes to be read from the rev!). But when it comes to reading from a work tree I don't think a tool that uses this library should say "create a .gitattributes file and read the gengo's usage." In onefetch's case, assuming we read from a work tree and not a rev, I believe the best thing is to allow onefetch to configure overrides itself. Here's the basic idea:
let file_source = Directory::new(".").with_overrides(overrides);
let gengo = Builder::new(file_source);
do_stuff_with(gengo.analyze());

Now onefetch is free to set overrides as it sees fit, whether it's from CLI options, a .onefetchrc file (or whatever we decide to call it), or something else.
cc @o2sh
Besides a fully generic implementation, I have been toying with the idea of a hybrid file source. The hybrid would be git-specific, use .gitattributes, and search down directories, taking advantage of .gitignore. That idea might alleviate your concerns, and I believe that this is what you had in mind by reading a git worktree. Yet, I'm still concerned about submodules. If .gitattributes is used to set the attributes of submodules, then we are, IMO, asking a user to make invalid .gitattributes. AFAIK .gitattributes cannot affect submodule contents for any of git's built-in attributes. I don't want to turn .gitattributes into our personal config file with additional, gengo-specific behavior. If git wouldn't do anything with the attribute's filepath, gengo shouldn't either.

As far as CLI usage goes, automatically detecting if an arg is a revspec or a directory is certainly a fun idea. However, I think there's a lot of risk for ambiguity. I often use branch names like test/javascript, for example, which looks a lot like a directory. Of course, we could check if a rev with that name exists, and if not try to read from a directory with that name. However, that magic can lead to confusion.

If I misspell and type tests/javascript, did I misspell a branch name or a directory? The error message would lose specificity ("tests/javascript is not a valid rev or directory"), and we wouldn't be able to just pass along errors returned by the library.
If the name matches both a revspec and a directory, what happens? Even git itself can't figure this out, and will require usage like git checkout tests -- tests to disambiguate.
Right now gengo can be run a specified directory (repository) with -R REPOSITORY and a specified rev with -r REV. This is a chance to disambiguate (previous point), but at the same time can complicate the CLI, as we'll then have to check for gengo FOO -r BAR -R BAZ.

Right now, at least, I don't really want to guess the user's intent, and would rather require explicitness. I also feel like subcommands are a great way to group CLI options that are only valid for specific usage.

So, just to summarize, here are my opinions:

When usage is tied to git, use git tools (git attributes, git ignore, git config), and act like git
When usage isn't tied to git, don't use git tools, and make behavior more configurable

First, keep in mind that any overrides should be the exception, not the rule. GitHub has about 40k .gitattributes with linguist-* attributes, and 132k .gitattributes without any linguist-* attributes. On top of that, the vast majority probably don't have any .gitattributes at all. So I don't think there will be a lot of wasted hard work if in some circumstances we don't read from .gitattributes.

I love these queries, thanks for bringing data to the discussion, it definitely helps. Following that line of argumentation, then .gitattribute support should also not be worth it for the git source though, especially since it's even less likely to see gengo-* attributes.

Ask users to put .gitattributes in non-git directories (Mercurial, SVN, or a plain old directory)

If we do this we'd also have to search up directories to find .gitattributes, with possibly no clear stopping point until /

Implement multiple ways to override (.gitattributes + config file) which leads to code complexity and questions about which should take precedence

Also, IMO setting overrides in .gitattributes is, TBH, a bit obscure. It's used for reading from a rev, and IMO the best way to do it since we can support reading from bare repos (thanks again for getting .gitattributes to be read from the rev!).

As a technical note, placing bare .gitattributes files would actually work even if no actual git repository is there if plumbing crates are used directly. And I was about to write the opposite until I was surprised by the flexibility of my own creation 😅.
But reading all of the above makes me think you don't actually want .gitattributes support at all, unless there is also a way to set overrides via API. This probably is due to the idea that it has to be generic.

Besides a fully generic implementation, I have been toying with the idea of a hybrid file source. The hybrid would be git-specific, use .gitattributes, and search down directories, taking advantage of .gitignore. That idea might alleviate your concerns, and I believe that this is what you had in mind by reading a git worktree. Yet, I'm still concerned about submodules. If .gitattributes is used to set the attributes of submodules, then we are, IMO, asking a user to make invalid .gitattributes. AFAIK .gitattributes cannot affect submodule contents for any of git's built-in attributes. I don't want to turn .gitattributes into our personal config file with additional, gengo-specific behavior. If git wouldn't do anything with the attribute's filepath, gengo shouldn't either.

I liked the previous idea of leaving submodule handling to the caller entirely. That way they can decide how to deal with .gitattributes, and whether or not the superproject's .gitattribute file gets to affect the submodule. Probably doing something like that isn't necessary either, and right now this train of thought lacks an anchor for a real-world usecase which makes it difficult to reason about. Thus, removing submodules from the equation would help.

Right now, at least, I don't really want to guess the user's intent, and would rather require explicitness. I also feel like subcommands are a great way to group CLI options that are only valid for specific usage.

I absolutely understand. Please note that all my comments aren't attempts to apply direction, but merely share my perspective, being well aware that everything comes with tradeoffs. As a user, I like gits behaviour of being fuzzy around its inputs while still being safe around ambiguity, while understanding that it's more complex on the implementation side. It's fair for gengo to be explicit and it won't be less usable for that.

So, just to summarize, here are my opinions:

When usage is tied to git, use git tools (git attributes, git ignore, git config), and act like git
When usage isn't tied to git, don't use git tools, and make behavior more configurable

I think its about making the decision if gengo has an opinion on the storage format of these overrides or not. If not, making the API for overrides based on callbacks is easy, but then each tool using gengo needs to sort this out by themselves. Most simply won't use overrides, maybe onefetch wouldn't or would implement them in terms of linguist which makes sense for compatibility (as much as possible).
The good thing is that one day, you could provide default implementations for these callbacks (or this API, to be less specific), and that's how users of gengo can opt-in to the default behaviour which unifies attribute handling, while also allowing them to add their own twist or maintain backwards compatibility to what was before.
This sounds so good to me that I'd even advocate for making overrides API based, possibly with a default implementation with .gitattributes if this is what you want.

Alright, thanks for the feedback!

I'd even advocate for making overrides API based, possibly with a default implementation with .gitattributes if this is what you want.

I hadn't thought too much about this, but this makes me think that something like Override::from_gitattributes could be the way to go. Right now I think I'm going to leave the current git-based implementation as a biased, special case, while other implementations are more open-ended. Frankly, I love the git-based behavior so much I'm reluctant to change it :laughing:
At a later point, I can consider making the git-based implementation more generic, after seeing how other implementations end up.

Frankly, I love the git-based behavior so much I'm reluctant to change it 😆
At a later point, I can consider making the git-based implementation more generic, after seeing how other implementations end up.

That sounds very feasible, and seems like a good way forward. Alternative implementations create facts which in turn make it easy to see which abstractions make sense to have more unification among them. Over all of this, I'd love it if it was clear what onefetch needs, as it's a candidate to be the first major 'client' if you will. Submodules in worktrees, yes or no, would be my primary question. And based on that answer one would know what should be done when looking at bare repositories as well. Maybe that was already clarified somewhere and I just forgot though 😅.


Preference / Settings page

Hi, I would like to see implementation of preference/settings page, and how to change textcolor / textbackground in important issues in preferences, and how to cope with style/themes according to preferences/settings.
androidx.preferenceFragmentCompat
https://developer.android.com/jetpack/androidx/releases/preference
RG

This is a good candidate for addition to Sunflower!  In the meantime, these are good resources for learning about Settings and Preferences:

Topic page on developer.android.com
Code sample


update vendored deps to latest remote version

Expected a flag for the godep update command, or a separate command, which would easily allow me to update all dependencies in my vendor dir, to the latest remote version, rather than the one in my $GOPATH/srcdir. I'm OK if the $GOPATH/src packages are updated because of this action, but right now godep update only updates to the local packages, which is a bit annoying.
Perhaps this is out of scope for this project, but right now I'm solving this by having a python script which goes through all dependencies listed in the Godeps.json manifest, to update all dependencies one by one using go get -u -d -v <DEP>. While this works, it does feel like a big hack.
Obviously my hack would also work, in case we would have specified a specific version (range) in the godeps dir, as I'm currently not parsing that myself. So I guess if we would support this feature, we should respect such specified limitations as well.
It's because I try to update all dependencies to the latest stable/master version every 2 to 4 weeks, so that these incremental updates never get too big. And usually they contain nice bug fixes, without really breaking the public apis, so it's nice to do them quite regularly. However, would be great if I don't require some hacky script to do so.
Is this currently already possible with godep? If not, how would I start on supporting this, as I'm ok to put dev time in for this one myself.
godep version output
$ godep version
godep v79 (darwin/amd64/go1.8)

go version output
$ go version
go version go1.8 darwin/amd64

I am closing this issue because the repo is going to be archived.


Add readonly and const Parsing in ts2mls

Related Issue: #134

I wonder if types like mut string acceptable. I made some changes in NewParser.scala because I found that we can do like this in OCaml:
val x : int ref = {contents = 42}

We will continue to work on this PR after this syntax's implementation:
class Foo { x: T; mut y: S; ... }


Fix #306 Enable tracking with enhanced trackers when adding to the library.

This PR is an attempt to fix the issue #306.
Tracking used to be enabled by default when adding an item to the library, which is not the case anymore.
This fix tries to bind with all logged In enhanced trackers upon adding an item to the library.
I think there still is an issue with the track being saved with a last_read_chapter value of 0.0, but the problem seems to lie in the SyncChapterProgressWithTrack and is not introduced by this fix.

Good job trying to fix your own issue

I think there still is an issue with the track being saved with a last_read_chapter value of 0.0, but the problem seems to lie in the SyncChapterProgressWithTrack and is not introduced by this fix.

Should be a separate PR

I think there still is an issue with the track being saved with a last_read_chapter value of 0.0, but the problem seems to lie in the SyncChapterProgressWithTrack and is not introduced by this fix.

Should be a separate PR

I moved this part to its new PR.


Invalid content /en/testnets/cardano/get-started/installing-and-running-the-cardano-node/building-the-node-from-source/

Where does the issue occur?
https://developers.cardano.org/en/testnets/cardano/get-started/installing-and-running-the-cardano-node/building-the-node-from-source/
Describe the issue
In the last paragraph the link "installing from source instructions" leads to a 404 page.
Expected changes
Correct broken link
Additional context

These pages have been updated. Thank you for your feedback


CORE-9865 Creation of InterOpFacade Processor - sending additional seed messages

Added 2 new seed messages to trigger search for and usage of InteropGroupPolicy 1) the hardcoded interop policy if found and the message is send to gateway, 2) the group policy id doesn't match and message is dropped.
Sample logs :
cd corda-bob-p2p-link-manager-worker-954749987-nnp22
cat logs.txt| grep -e Dropping -e Sending -e "discarded."  -e "interop" -b3 | cut -d] -f3-

The logs were further trimmed of class name et just to show log message.
An outbound message is match with interoperability group (there  is not "normal" group and there is no VitualNode):
Could not get virtual node info for holding identity [HoldingIdentity(x500Name=CN=Alice, O=Alice Corp, L=LDN, C=GB, groupId=3dfc0aae-be7c-44c2-aa4f-4d0d7145cf08)]
Could not get CPI metadata for holding identity [HoldingIdentity(x500Name=CN=Alice, O=Alice Corp, L=LDN, C=GB, groupId=3dfc0aae-be7c-44c2-aa4f-4d0d7145cf08)] and CPI with identifier [null]. Any updates to the group policy will be processed later.
Searching for interoperability group for HoldingIdentity(x500Name=CN=Alice, O=Alice Corp, L=LDN, C=GB, groupId=3dfc0aae-be7c-44c2-aa4f-4d0d7145cf08)
Sending outbound message 2 to {"x500Name": "CN=Alice from Other Cluster, O=Alice Corp, L=LDN, C=GB", "groupId": "3dfc0aae-be7c-44c2-aa4f-4d0d7145cf08"} for http://localhost:8080.

Neither there is no VirtualNode, nor matching interop group policy id (groupId=non-existing-group), the message is dropped:
Could not get virtual node info for holding identity [HoldingIdentity(x500Name=CN=Alice, O=Alice Corp, L=LDN, C=GB, groupId=non-existing-group)]
Could not get CPI metadata for holding identity [HoldingIdentity(x500Name=CN=Alice, O=Alice Corp, L=LDN, C=GB, groupId=non-existing-group)] and CPI with identifier [null]. Any updates to the group policy will be processed later.
Searching for interoperability group for HoldingIdentity(x500Name=CN=Alice, O=Alice Corp, L=LDN, C=GB, groupId=non-existing-group)
Could not get interop group policy for holding identity [HoldingIdentity(x500Name=CN=Alice, O=Alice Corp, L=LDN, C=GB, groupId=non-existing-group)].
Could not find the group information in the GroupPolicyProvider for HoldingIdentity(x500Name=CN=Alice, O=Alice Corp, L=LDN, C=GB, groupId=non-existing-group) to {"x500Name": "CN=Alice Alter Ego, O=Alice Alter Ego Corp, L=LDN, C=GB", "groupId": "non-existing-group"}.
   The message 2 was discarded.

Awaiting https://github.com/corda/corda-runtime-os/pull/3268 to be merge first


Version 8.0.4c changed to 8.0.4.c

Version 8.0.4c changed to 8.0.4.c in accordance to winget-cli Version sorting

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?
[x] This PR only modifies one (1) manifest
[X] Have you validated your manifest locally with winget validate --manifest <path>?

[X] Does your manifest conform to the 1.5 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

Proposed fix for https://github.com/microsoft/winget-pkgs/issues/122931

I have redone the PR herer: https://github.com/microsoft/winget-pkgs/pull/122942


Provision a demo instance for weekly demos in staging envrionment

Everybody will be creating and deleting VMs all the time which is a ton of repeatable time every week, instead, we will just have one reusable VM and #2527 for simple demos.

Done in c4a3c2a630aabfe25ad4f27e9769cd7b8884b265


Implement TextField component

Description

Implement a TextField component with more styles in the second theme. So that user can enter his/her details in the text field.
Reference: https://www.figma.com/file/rX1jl4pa04VD9itnzneeih/Webiu?node-id=0%3A2
Changes proposed in the pull request

Implement StyledSelect component
Implement the story for the component
Create documentation for the component

Screenshots (In case of UI changes):

Types of changes

[x] New feature (non-breaking change which adds functionality)

Checklist:

[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] All new and existing tests have passed and it does not give any unexpected error for the same.

Steps to test

npm i
npm run storybook (story will be displayed in CustomerService/StyledTextField )

Resolves https://github.com/scorelab/Webiu/issues/267

@charithccmc @dinithminura can you review the PR please :)


runtime: TestStackBarrierProfiling failing on mipsle

linux-mipsle at 1b66b38e25af567aa30a7b22581c05285be3564a
https://build.golang.org/log/59970800b3d64dc2d3d4231a7de764086a67ac15
--- FAIL: TestStackBarrierProfiling (6.99s)
	pprof_test.go:374: subprocess failed with exit status 1:
		--- FAIL: TestStackBarrierProfiling (6.92s)
			pprof_test.go:193: total 2 CPU profile samples collected
			pprof_test.go:205: too few samples; got 2, want at least 5, ideally 20
			pprof_test.go:150: retrying with 400ms duration
			pprof_test.go:193: total 8 CPU profile samples collected
			pprof_test.go:205: too few samples; got 8, want at least 10, ideally 40
			pprof_test.go:150: retrying with 800ms duration
			pprof_test.go:193: total 15 CPU profile samples collected
			pprof_test.go:205: too few samples; got 15, want at least 20, ideally 80
			pprof_test.go:150: retrying with 1.6s duration
			pprof_test.go:193: total 31 CPU profile samples collected
			pprof_test.go:205: too few samples; got 31, want at least 40, ideally 160
			pprof_test.go:150: retrying with 3.2s duration
			pprof_test.go:193: total 33 CPU profile samples collected
			pprof_test.go:205: too few samples; got 33, want at least 80, ideally 320
		FAIL
FAIL
FAIL	runtime/pprof	11.166s

/cc @vstefanovic @cherrymui

Is it due to issue #13405, in particular, https://github.com/golang/go/issues/13405#issuecomment-168884316 ?
$ gomote run user-cherryyz-linux-mipsle-0 /bin/zcat /proc/config.gz | grep 'CONFIG_HZ\|HIGH_RES'
# CONFIG_HZ_24 is not set
# CONFIG_HZ_48 is not set
CONFIG_HZ_100=y
# CONFIG_HZ_128 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_256 is not set
# CONFIG_HZ_1000 is not set
# CONFIG_HZ_1024 is not set
CONFIG_HZ=100
CONFIG_HZ_PERIODIC=y
# CONFIG_HIGH_RES_TIMERS is not set

/cc @aclements

@cherrymui, good find. Seems likely.
Should we make that test do the /bin/zcat /proc/config.gz and skip if it doesn't find CONFIG_HIGH_RES_TIMERS set?

/cc @vstefanovic

Since stack barriers aren't even on by default now, I'm sure this is the timing issue. (I'll be removing this test in Go 1.9 when I remove support for stack barriers.)
I'm not positive, but I think CONFIG_HZ_100 would actually be fine if CONFIG_HIGH_RES_TIMERS were set.

@bradfitz, just found http://elinux.org/High_Resolution_Timers#How_to_detect_if_your_timer_system_supports_high_resolution
maybe grep resolution /proc/timer_list is a better way to find out timer resolution (given that kernel config may not always available)?

@cherrymui, feel free to improve it. I don't know exactly how to parse /proc/timer_list. I'm just trying to get build.golang.org to not be so red.


[BUG]: --deck has no effect in streamdeckc

In raising this issue I confirm that

[X] I have fully completed the issue template
[X] I have searched open and closed issues for duplicates
[X] I have read the Contribution Guidelines
[X] I have read the Code of Conduct
[X] I have read the Documentation

Describe the bug
Hey everyone :-) I'm trying to control the state of a button with a script, however the --deck argument doesn't seem to work when I include it in the command.
If I execute streamdeckc -a SET_STATE -p 0 -b 0 -s 1 then the commands works, however if I execute streamdeckc -a SET_STATE -d 1 -p 0 -b 0 -s 1 it no longer works. I have three streamdecks connected, so I tried -d 0, -d 1, -d 2 and -d 3, all without effect, and checked the JSON that can be exported and it doesn't show any additional or empty streamdecks.
EDIT: It seems to only issuing the command to last opened device in the GUI, even if it is shut.
Any suggestions would be appreciated, as I need this feature to work for properly updating button states of my microphones when Twitch streaming.
Oh, and as an unrelated aside, you can add Hyprland to the tested desktop environments, well... window managers, listed in the ReadMe :-)
Steps to reproduce the behavior
Execute:
streamdeckc -a SET_STATE -d 0 -p 0 -b 0 -s 1
Expected behavior
SET_STATE successfully.
Screenshots
No response
System Information
Arch
Hyprland
Python 3.11.6
Stream Deck Version
Mini x3

Thankyou <3


Adding a basic error page

What path is actually serving up this file through the spring boot server?

There's an error handler that writes this out...not like any kind of redirect.  Spring Boot magic again!


Servicemonitor should have ability to pass additional labels to the resource

I would like to be able to add additional Kubernetes labels to the ServiceMonitor as this is often required by the Prometheus operator to identify the Service and pod(s).
Suggested updates for the servicemonitor.yaml
{{- if .Values.servicemonitor.enabled }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: {{ template "atlantis.fullname" . }}
  labels:
{{- include "atlantis.labels" . | nindent 4 }}
  {{- with .Values.servicemonitor.additionalLabels }}
    {{- toYaml . | nindent 4 }}
  {{- end }}
  {{- with .Values.service.annotations }}
  annotations:
{{ toYaml . | indent 4 }}
  {{- end }}
...

And updating the values/yaml also:
servicemonitor:
  # to enable a Prometheus servicemonitor, set enabled to true,
  #   and enable the metrics in this file's repoConfig
  #   by setting a value for metrics.prometheus.endpoint
  enabled: false
  interval: "30s"
  path: /metrics
  # Prometheus ServiceMonitor labels
  additionalLabels: {}

Implemented by https://github.com/runatlantis/helm-charts/pull/354.


Add a readme

Addition of a readme
With the project becoming bigger and bigger, a readme is quite necessary to have, don't you think?

yes a readme is necessary, you can open a pr on it

can you assign me this issue?

contributions are most welcome!

readme added.


Add section about Branding to README

Hey @isabellewei Can you please let us know what we need to test here from front end as well as back end side from our end ?
Thanks..!!

@Celoqa I've confirmed the branding section is updated in the README.md and that the branding repo is working as intended. No need for additional testing.

Thanks @MuckT or the update.


[Feature]: Full rootless Docker support

Description
Support full rootless Docker. On localhost (coolify instance) and on connected remote servers.
This will greatly improve the security of the coolify containers and the services running on them, as rootless Docker isolation is required to truly separate each container.
cc @DeVoresyah
Minimal Reproduction (if possible, example repository)

Exception or Error
No response
Version

Good article: https://medium.com/@mccode/processes-in-containers-should-not-run-as-root-2feae3f0df3b

This should be possible with Docker.
A second thing to consider is implementing podman for better security, or even as an addition to docker, since it runs more securely by default (rootless). https://podman.io/

Some more help to implement:
To run Docker as a router, we need to use custom Docker images for all services (so we essentially need to rewrite the services a bit) as we need to use the yml file and the a custom dockerfile for each service:
Issue that adds this: https://github.com/coollabsio/coolify/issues/2552

In each service folder there should be a service.yml file and a custom service.dockerfile, so for example for WordPress: wordpress.yml and DOCKERFILE inside the wordpress service folder.
In the custom docker file we use the flowing command to create a random user that does not have root privileges and disable the root user:

RUN groupadd -r <user> && useradd -r -g <group> <user>
RUN chsh -s /usr/sbin/nologin root

In the yml file we access the image something like this:

service:
    build:
      context: .

@andrasbacsai if you need more help just let me know.

+1


fix references to support objects when using DI

some change to how the support objects are initialized removed the original
reference to the support object and replaced it a "cloned" object. However,
since Object.assign was used, not all properties have been copied and especially
the original reference was lost.
This implementation uses a proxy object to load the injected modules on demand
and returns the original reference to the module. This should fix all usages
of support objects, where the type is not a plain object.
Fixes: #1679

Thank you! I also wanted to use Proxy for that but was not so smart to implement in a correct way

Concerning the broken tests for webdriver, this seems to be due to an upgrade to latest chrome in the selenium standalone images. Therefor I currently have the docker images fixed to selenium/standalone-chrome:3.141.59-oxygen, where I.doubleClick is still working for instance. So I guess this has nothing to do with the change of this PR.

Sure, it's nothing to do with this PR.
If you know how to fix the builds, could you please update .travis.yml and send a patch?
Much appreciated

The day has come. I'm merging!

@johnyb your implementation broke recursive loading of support objects. Proxy was good but still, we need to initialize objects and then use Object.assign. In a new implementation https://github.com/Codeception/CodeceptJS/blob/codecept-221-release/lib/container.js#L213

I still use proxy to load objects on demand
I use Object assign but only for real objects
I added test for loading objects as an array
I fixed https://github.com/Codeception/CodeceptJS/issues/1795
I added a test from @Vorobeyko to the core.

Very good, thanks!


Removed unused internals from the generated C#

@tritao do not merge this, it can be further optimised.


ResponseCashingMiddleware Uses IMemoryCache

Description
In the article "Troubleshooting" section at this location https://learn.microsoft.com/en-us/aspnet/core/performance/caching/middleware?view=aspnetcore-8.0#troubleshooting it`s mentioned that The Response Caching Middleware  uses IMemoryCache,
But when I add this "on-the-fly" middleware right before the ResponseCashingMiddleware trying to inject an instance of service IMemoryCache it throwed an exception indicating it`s not registered on the container, meaning the ResponseCashingMiddleware did not register it nor using it.
builder.Services.AddResponseCaching();

.......

app.Use(async (c, r) =>
{
    var cache = c.RequestServices.GetRequiredService<IMemoryCache>();

    await r(c);
});

app.UseResponseCaching();

Can you please explain if I am getting it wrong, whether the middleware is using IMemoryCache or not as this is confusing, as It`s also mentioned by the article that the developer cannot invalidate cached responses using the ResponseCashingMiddleware  BUT if you are saying it uses IMemoryCache and if somehow I can inject ,then we do can invalidate a specific cached request!
Thanks a lot for your patience.
Kind regards
Page URL
https://learn.microsoft.com/en-us/aspnet/core/performance/caching/middleware?view=aspnetcore-8.0#troubleshooting
Content source URL
https://github.com/dotnet/AspNetCore.Docs/blob/main/aspnetcore/performance/caching/middleware.md
Document ID
13154131-c2b6-2c72-fb0e-1d83bca4e21a
Article author
@tdykstra

Hello! Thanks for contacting us. This sounds like a general question about using ASP.NET Core. While we try to look at and respond to all issues, for questions like this we recommend posting to a community support group like Stack Overflow with the asp.net-core tag.

Stack Overflow
ASP.NET Core Slack
ASP.NET Gitter

However, it sounds like your goal is more control over caching at the server -- that is one of the reasons you might want to choose output caching over response caching.   See this article: https://learn.microsoft.com/en-us/aspnet/core/performance/caching/overview?view=aspnetcore-8.0#output-caching


Annotation table gives create error when using objectid

If I try to create an incident to an annotation row then I get the error:
Expected non empty guid
Parameter: objectId
And the weird thing is that it worked before.

Would need a bit more to work with here... :)
can you give me an example of the code that is doing this? and what is no longer working?
thanks.


Add Dockerfile for registry, part of #15, fixes #40

Description
Add Dockerfile to create docker container for model-registry.
Add instructions in README.md for building and using the docker container.
How Has This Been Tested?
Commands documented in the README.md were used to start the server, and the test client was run against the container server successfully.
Merge criteria:

[X] The commits and have meaningful messages; the author will squash them after approval or will ask to merge with squash.
[X] Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
[X] The developer has manually tested the changes and verified that the changes work

The builder pattern is a common Docker pattern to avoid polluting the final container image with project source and build tools. If the builder image fails, the docker build itself will fail in CI.

I believe I understand the pattern.
What I'm saying is that CI testing of the docker build would be helpful to ensure reproducibility and avoid potential regressions, noted as https://github.com/opendatahub-io/model-registry/issues/50


Accepting bad indexes

When a bad index (e.g. an index as an array) passed to a method, instead of checking it to be either a string or an integer, a TypeError exception is thrown. Instead, a custom exception should be thrown.

Sorry! Moved here.


Add services

Add services for the following API commands. Will require some updates to the python-fullykiosk library as well.

[ ] /?cmd=loadStartUrl&password=[pass]

[ ] /?cmd=loadUrl&url=[url]&password=[pass]

[ ] /?cmd=restartApp&password=[pass]

[ ] /?cmd=startApplication&package=[pkg]&password=[pass]

[ ] /?cmd=toForeground&password=[pass]

[ ] Generic set configuration service
/?cmd=setBooleanSetting&key=[key]&value=[true|false]&password=[pass]
/?cmd=setStringSetting&key=[key]&value=[value]&password=[pass]

I think entity services will be the easiest, I haven't been able to come up with a good way to use general services and specifying which config entry to target. I'm leaning towards the media_player entity being the best target for most of the services... unless there's a way we can use entity services  that target any entity of a config entry?

I just looked at how the Hue component does this for example, and the idea seems to be that any general service runs on all config entries. There also doesn't seem to be a way to determine the config entry, so I guess that means the services have to be defined on a single entity :/ I was favoring the media_player entity for this as well, so I'll add them there 👍

I've added almost all services that are in this list. I've created a PR for the python-fullykiosk project to support setBooleanSetting and setAudioVolume. Once that is approved I can finish the last bits.

PR #22 merged and services added 😃  Thanks again @bjw-s !


Book Recommendation page [FEATURE REQUEST]

Spiritual Book Recommendations page, offering handpicked soul-enriching books on mindfulness, self-discovery, and ancient wisdom.

hey @akshitagupta15june please assign me this task under iwoc'24.


2.2.4 deletes pid file while script is still running

Since version 2.2.4 this library is broken. If you run the same script twice, you get the pid.PidFileAlreadyLockedError: [Errno 35] Resource temporarily unavailable error, however the pid file is also deleted and if you run the script again it will succeed even though the first instance is still running.
Below is a sample script. Run it in one terminal, then run it again twice in a separate terminal to observe the incorrect behavior.
#!/usr/bin/env python

from pid import PidFile
from time import sleep

if __name__ == '__main__':
	with PidFile('test123', piddir='/tmp'):
		sleep(9999999)

WoW thats bad. We really need to catch that in our tests. Do you think you could create a PR for a test for this ?
(Otherwise I will try to make one as soon as I can)

I created a PR for a test. It's not very elegant but I couldn't figure out a better way since it needs to spawn a separate interpreter so that the atexit handler is run before the first instance finishes.

Could you please test with ce9a6e11438fb168d94c522a44f86cdd047271f4 ?

Awesome !
2.2.5 is released on https://pypi.org/project/pid/ :-)


Drive file getName() method not defined?

I am reading in Google Drive links to images from a Google sheet. The links contain the file IDs and the images are shared to anyone with the link. I am able to retrieve and download the images without a problem.
However I would like to name the images with the same filename that they have in the Google Drive folder, but using the getName() method doesn't return the file's name and throws the error
Fatal error:  Uncaught Error: Call to undefined method GuzzleHttp\Psr7\Response::getName()

My code is as follows:
$service = new Google_Service_Drive($client);
$file = $service->files->get( $fileID, array("alt" => "media") );
echo $file->getName();

I seem to be missing something to be able to get the filename.

It would be really nice if you explained how did you solve it before closing the issue 😃
It would be a great help to me as I'm facing the same problem.

I pulled the name out in a slightly different way:
$result = $service->files->get($fileID, ['fields' => 'name,md5Checksum,size']);
$fn = $result->name;


Frequent crashes on Windows with Assertion failed: IDAndOffset.first.isValid() && "invalid FileID for transition"

Assertion failed: IDAndOffset.first.isValid() && "invalid FileID for transition", file C:\a\swift-build\swift-build\SourceCache\llvm-project\clang\lib\Serialization\ASTReader.cpp, line 6609

The assertion in question: https://github.com/apple/llvm-project/blob/51859f9280134994d482f8420619760409903585/clang/lib/Serialization/ASTReader.cpp#L6675
We're seeing this happen frequently enough to hit the 6 crashes in 3 minutes no-restart limit.
This may not be actually be a sourcekit-lsp issue, but as it materializes as a crash I figured it was worth flagging here.

Tracked in Apple’s issue tracker as rdar://125018783

Do you have a crash trace of the call stack that caused the assertion failure?


Add seaborn-data datasets

I'd like to add the dataset of seaborn-data so that I can use this in examples of charty.

Thanks!


Yara 2.x-3.x+ all crash, internal error 25, on large rules (example inlined)

We auto-generate numerous yara rules with a high volume of strings in order to utilize yara as a large-scale pattern-matching and cluster-identifying tool. All versions above yara 1.7 seem to crash on large rules with this "internal error: 25". Because of this bug, we cannot move from yara 1.7 for our rulesets without sacrificing this project.
I have an example of an offending rul (this one built with junk data) to my github repo at:
https://github.com/ckane/yara23-bug-rules

You're overflowing the stack used by YARA. You can work around it by bumping STACK_SIZE in libyara/exec.c to a much bigger number. I used 85000, just because it was bigger than the number of strings you have. Just a quick read through the code, so I could be completely wrong here, but I think when you use 'any of them' in a condition it will push every non-chained string onto the stack. Since the stack size is much smaller than the number of things being pushed you get the overflow.
As to a possible better answer than bumping STACK_SIZE, I really don't know off the top of my head.

Wesley is right, when you use "any of them" a reference to each string is pushed in the stack, with that many strings your overflowing the stack. One solution is setting a higher STACK_SIZE as suggested, another one is splitting this huge rule in multiple smaller rules.
I'm going to mark this as "won't fix" because this is not an actual bug. You're just hitting an internal limit.

For my particular application, splitting the strings into separate rules is likely a no-go. I'll try increasing the stack size, as I have "all the RAM" on the environment doing the work, and see what that nets me. Just curious however, are you aware as to whether there's a ceiling to the stack size? In one case I am seeing over 200,000+ strings in a single rule.
From your description of the implementation, it sounds to me like this is a per-rule limit, though? Is there perhaps a way to provide a command-line argument to modify this at run-time?

Ok I set the stack size to 800000 and that seems to work. However, if I bump it to 1048576 (my first attempt before dropping incrementally to lower sizes), I get a seg fault:
Program received signal SIGSEGV, Segmentation fault.
0x000000000043159b in yr_execute_code (rules=<error reading variable: Cannot access memory at address 0x7fffff7fd598>,
    context=<error reading variable: Cannot access memory at address 0x7fffff7fd590>,
    timeout=<error reading variable: Cannot access memory at address 0x7fffff7fd58c>,
    start_time=<error reading variable: Cannot access memory at address 0x7fffff7fd580>) at exec.c:130
130     {
(gdb) bt
#0  0x000000000043159b in yr_execute_code (rules=<error reading variable: Cannot access memory at address 0x7fffff7fd598>,
    context=<error reading variable: Cannot access memory at address 0x7fffff7fd590>,
    timeout=<error reading variable: Cannot access memory at address 0x7fffff7fd58c>,
    start_time=<error reading variable: Cannot access memory at address 0x7fffff7fd580>) at exec.c:130
#1  0x0000000000413f19 in yr_rules_scan_mem_blocks (rules=0x655010, block=0x0, flags=0, callback=0x403c99 <callback>, user_data=0x7fffffffe17d, timeout=1000000)
    at rules.c:425
#2  0x0000000000414189 in yr_rules_scan_mem (rules=0x655010, buffer=0x7ffff7f2e000 "\177ELF\002\001\001", buffer_size=692295, flags=0,
    callback=0x403c99 <callback>, user_data=0x7fffffffe17d, timeout=1000000) at rules.c:515
#3  0x00000000004141eb in yr_rules_scan_file (rules=0x655010, filename=0x7fffffffe17d "yara", flags=0, callback=0x403c99 <callback>, user_data=0x7fffffffe17d,
    timeout=1000000) at rules.c:540
#4  0x0000000000404880 in main (argc=2, argv=0x7fffffffde98) at yara.c:1043
(gdb)

So for the time being it would seem that the lower number will work for my purposes, although the size of my dataset only increases with every passing month, so at some point I'll probably end up hitting the wall again.
Thanks for the quick response and work-around.

I know this is a wontfix but there is something about that segfault that is interesting. I went ahead and duplicated this on an OS X (10.10.2) machine. I then went looking for what would cause it and didn't come up with anything in the source code. Stepping through it in a debugger on OS X I noticed this is the start of the disassembly for yr_execute_code:
(lldb) dis -F intel -c 11
yara`yr_execute_code at exec.c:130:
-> 0x10002af60:  push   rbp
   0x10002af61:  mov    rbp, rsp
   0x10002af64:  sub    rsp, 0x8005a0
   0x10002af6b:  mov    rax, qword ptr [rip + 0x2e08e] ; (void *)0x00007fff7ac96070: __stack_chk_guard
   0x10002af72:  mov    rax, qword ptr [rax]
   0x10002af75:  mov    qword ptr [rbp - 0x8], rax
   0x10002af79:  mov    qword ptr [rbp - 0x8004b8], rdi
   0x10002af80:  mov    qword ptr [rbp - 0x8004c0], rsi
   0x10002af87:  mov    dword ptr [rbp - 0x8004c4], edx
   0x10002af8d:  mov    qword ptr [rbp - 0x8004d0], rcx
   0x10002af94:  mov    dword ptr [rbp - 0x8004d4], 0x0

The crash happens executing the instruction at 0x10002af79.
I then tried to reproduce this on a Linux machine and two FreeBSD machines. The crash did not happen on one of my FreeBSD machines, but did on the Linux and the other FreeBSD machine.
This is the disassembly from yr_execute_code on the Linux machine:
(gdb) disass
Dump of assembler code for function yr_execute_code:
   0x0000000000430fc8 <+0>:	push   rbp
   0x0000000000430fc9 <+1>:	mov    rbp,rsp
   0x0000000000430fcc <+4>:	push   rbx
   0x0000000000430fcd <+5>:	sub    rsp,0x800538
=> 0x0000000000430fd4 <+12>:	mov    QWORD PTR [rbp-0x800518],rdi
   0x0000000000430fdb <+19>:	mov    QWORD PTR [rbp-0x800520],rsi
   0x0000000000430fe2 <+26>:	mov    DWORD PTR [rbp-0x800524],edx
   0x0000000000430fe8 <+32>:	mov    QWORD PTR [rbp-0x800530],rcx
   0x0000000000430fef <+39>:	mov    DWORD PTR [rbp-0x800510],0x0

And this is the output from yr_execute_code on the NOT WORKING FreeBSD machine (this one crashes at 0x439333):
(gdb) disass yr_execute_code
Dump of assembler code for function yr_execute_code:
0x0000000000439300 <yr_execute_code+0>: push   rbp
0x0000000000439301 <yr_execute_code+1>: mov    rbp,rsp
0x0000000000439304 <yr_execute_code+4>: sub    rsp,0x800580
0x000000000043930b <yr_execute_code+11>:        mov    QWORD PTR [rbp-0x10],rdi
0x000000000043930f <yr_execute_code+15>:        mov    QWORD PTR [rbp-0x18],rsi
0x0000000000439313 <yr_execute_code+19>:        mov    DWORD PTR [rbp-0x1c],edx
0x0000000000439316 <yr_execute_code+22>:        mov    QWORD PTR [rbp-0x28],rcx
0x000000000043931a <yr_execute_code+26>:        mov    DWORD PTR [rbp-0x4b4],0x0
0x0000000000439324 <yr_execute_code+36>:        mov    rcx,QWORD PTR [rbp-0x10]
0x0000000000439328 <yr_execute_code+40>:        mov    rcx,QWORD PTR [rcx+0x8]
0x000000000043932c <yr_execute_code+44>:        mov    QWORD PTR [rbp-0x4c0],rcx
0x0000000000439333 <yr_execute_code+51>:        mov    DWORD PTR [rbp-0x800514],0x0

The WORKING FreeBSD machine:
(gdb) disass
Dump of assembler code for function yr_execute_code:
0x000000000043a320 <yr_execute_code+0>:	push   rbp
0x000000000043a321 <yr_execute_code+1>:	mov    rbp,rsp
0x000000000043a324 <yr_execute_code+4>:	sub    rsp,0x800580
0x000000000043a32b <yr_execute_code+11>:	mov    QWORD PTR [rbp-0x10],rdi
0x000000000043a32f <yr_execute_code+15>:	mov    QWORD PTR [rbp-0x18],rsi
0x000000000043a333 <yr_execute_code+19>:	mov    DWORD PTR [rbp-0x1c],edx
0x000000000043a336 <yr_execute_code+22>:	mov    QWORD PTR [rbp-0x28],rcx
0x000000000043a33a <yr_execute_code+26>:	mov    DWORD PTR [rbp-0x4b4],0x0

Could this be some weird compiler issue causing it to incorrectly calculate the offset from rbp? I'm clearly in over my head, but I thought I would at least share what I've found. Here's the information for each machine:
OS X:
wxs@psh yara % gcc -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 6.0 (clang-600.0.56) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin14.1.0
Thread model: posix
wxs@psh yara %

Linux:
wxs@foo:~/yara$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.2 LTS
Release:	14.04
Codename:	trusty
wxs@foo:~/yara$ gcc --version
gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

wxs@foo:~/yara$

FreeBSD (NOT WORKING):
wxs@syn yara % uname -r
10.1-RELEASE-p5
wxs@syn yara % cc -v
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Target: x86_64-unknown-freebsd10.1
Thread model: posix
Selected GCC installation:
wxs@syn yara %

FreeBSD (WORKING):
wxs@ack yara % uname -r
11.0-CURRENT
wxs@ack yara % cc -v
FreeBSD clang version 3.5.1 (tags/RELEASE_351/final 225668) 20150115
Target: x86_64-unknown-freebsd11.0
Thread model: posix
wxs@ack yara %

So I suspect this has something to do with how the compiler allocates space on the stack. In the yr_execute_code(...) function, stack is a local stack-allocated array variable sized at STACK_SIZE*sizeof(union STACK_ITEM). I wonder if this is a matter of one of the stack-smashing-protections or event RLIMIT_STACK getting in the way...though it is an odd way to fail.
I tried -fno-stack-protector and it still failed.
Then I just converted:
union STACK_ITEM stack[STACK_SIZE];

To:
union STACK_ITEM *stack;

in order to allocate it on the heap instead (and of course added free(stack) in the places where it belongs in the function. Making that change makes the crash go away, and I doubt that there's any significant performance penalty. On a good note, as well, that change probably could eliminate a potential future (but untested) vulnerability vector for a piece of software that's widely known to be adopted by the world's security community, and who's purpose is to flow "untrusted" and "suspected attack" data through a complicated pipeline of C code.

That would have the additional benefit of enabling the stack-size to be converted into a run-time tunable.

--- libyara/exec.c.old  2015-02-19 23:33:52.494000000 -0500
+++ libyara/exec.c      2015-02-19 23:34:06.263000000 -0500
@@ -133,7 +133,7 @@
   int32_t sp = 0;
   uint8_t* ip = rules->code_start;

-  union STACK_ITEM stack[STACK_SIZE];
+  union STACK_ITEM *stack;
   union STACK_ITEM r1;
   union STACK_ITEM r2;
   union STACK_ITEM r3;
@@ -156,6 +156,8 @@
   clock_t start = clock();
   #endif

+  stack = calloc(STACK_SIZE, sizeof(union STACK_ITEM));
+
   while(1)
   {
     switch(*ip)
@@ -164,6 +166,7 @@
         // When the halt instruction is reached the stack
         // should be empty.
         assert(sp == 0);
+        free(stack);
         return ERROR_SUCCESS;

       case OP_PUSH:
@@ -1043,8 +1046,10 @@

       if (++cycle == 10)
       {
-        if (difftime(time(NULL), start_time) > timeout)
+        if (difftime(time(NULL), start_time) > timeout) {
+          free(stack);
           return ERROR_SCAN_TIMEOUT;
+        }

         cycle = 0;
       }
@@ -1056,5 +1061,6 @@
   // After executing the code the stack should be empty.
   assert(sp == 0);

+  free(stack);
   return ERROR_SUCCESS;
 }

I was thinking something similar (move the stack off the stack). If you are going to send a PR you will want to use yr_alloc and yr_free though. I have no clue what Victor will say about this though.

 Might also take a stab at cmdline option... all dependent upon what other work I need to procrastinate on.

I'd probably suggest moving stack into the YR_CONTEXT struct, in that event and initialize it in libyara/rules.c where needed, enabling fewer malloc/free cycles and a cleaner approach to memory management than my first version above. Intuitively the "stack" following around the execution context would probably be the expected behavior to a new observer.

I think it's a good idea to use dynamically allocated memory for the stack. Maintaining the pointer in YR_CONTEXT as ckane suggests seems like a elegant solution.

Cool - I'll work on a patch this weekend, time permitting

is the weekend terminated? or this issue is fixed in another way? I still have it today.

I think it was merged in. You will need to provide more context (preferably in another issue) with what you're seeing. Please include a rule which triggers the problem for you.
You may also want to look at the -k argument to yara too, as I think that is directly related to the work @ckane did in this area.


Error when run pokecli.py

$ python pokecli.py
File "pokecli.py", line 70
print "Run following command to get latest update: pip install -r requirements.txt --upgrade"
^
SyntaxError: Missing parentheses in call to 'print'

Seem you're using Python 3 executable while code you posted has a Python 2 syntax.

closing


Manual update of one page summary last bullet

Once the decision table is established these number need to be manually updated

closing...


resolves #126 convert styles to SCSS

closes #114
closes #115
closes #127

Now we can see all fifty shades of gray we use!

So, the plan is to use sass gem for now. It looks like sassc got deprecated too with libsass itself: https://sass-lang.com/blog/libsass-is-deprecated
We were not going to use it anyway because of issues under JRuby. Currently supported implementation is sass-embedded gem, but that needs Ruby 2.6+, a bit too modern for us.


Allow cinderella components to detect whether ExtData can provide a field

The idea is that a cinderella component would only compute a field if the field could not be provided by ExtData.    This is directly opposite of the initial intent of ExtData and will require some thought.

It can be difficult to accomplish this for all the fields imported in the Cinderella Component (CC). In my point of view, we can decide to do it for a selected number of fields. The CC imports a set of fields to accomplish at least the following tasks:
(1) Compute the mass fluxes and courant numbers
(2) Derive other meteorological related variables
(3) Export meteorological related variables for outputting only.

In my point of view, this project should focus on Task 1 only. Task 2 and Task 3 can be handled using a flag. As a matter of fact, Task 3 is done using a flag.


Use templates for the value type

Fixes #8

Thanks!
The code looks good, I´ll try it tonight when I come home and give you an update

No problem :+1:


Deploying the encoder service

Had a quick chat with @ingalls and @batpad. We'll use a basic ECS cluster with 2 autoscaling groups:

1 for CPU that's always-on
another one that's GPU backed but by request
For the first pass we'll have a manual process to increase the min size of the autoscaling group
Later this can be managed via potentially a simple API gateway endpoint that modifies the min size on the cluster

@ingalls — @rbavery has done pretty much all the work in terms of dockerizing these in this repo. gpu and cpu. So you should be able to pretty much take that and setup ECS.
I think we could combine this to one image if you need for simplicity. @rbavery would that be ok or are there other considerations?
cc @srmsoumya

Talked with @geohacker but reiterating here, I separated the GPU encoder service thinking that would be run in an on-demand fashion to keep cost down. Whenever new imagery becomes available, we'd spin up the GPU encoder and then spin it down once we have the embeddings.
The CPU decoder would get used more frequently and consistently on top of the embeddings that were generated, so I was thinking it should be it's separate service that is always running in the backend for something like PEARL.

This is deployed, closing!


[FEATURE]  Expect to support the filesystem not implementing the append-mode.

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before asking

[X] I have searched in the issues and found no similar issues.

Describe the feature
Cloud object storage and its corresponding accelerated cache are widely used in storage-computing separation architecture.
But many of them may not implement the append-mode or the write amplification of the append-mode will be tricky.
Is it possible to consider supporting the way of independently storing each block as a data file at the same time?
Motivation
Cloud object storage and its corresponding accelerated cache are widely used in storage-computing separation architecture.
But many of them may not implement the append-mode or the write amplification of the append-mode will be tricky.
Describe the solution
The initially envisaged solution is to implement a non-append abstract storage type, such as: AbstractObjectStorageWriteHandler, AbstractObjectStorageReadHandler, and AbstractObjectDeleteHandler,  and Implement the basic interface required by RSS in it.
The implementation of the read, write, and delete interfaces placed in the specific storage layer can be implemented by subclasses.
Additional context
No response
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Do you mean that one dataFlushEvent could directly write a new single oss file? If so, how to maintain the index file?

Do you mean that one dataFlushEvent could directly write a new single oss file? If so, how to maintain the index file?

Aren't several blocks appended to a data file at present?  I want to write a block into a single data file, so it corresponds to an index file.

Could u help give a data layout picture, which includes the relations of data block and oss file name.
Besides, we should also consider how to support the local_order in object store.

Could u help give a data layout picture, which includes the relations of data block and oss file name.
Besides, we should also consider how to support the local_order in object store.

Okay, I will provide a design gram to describe it later.

Aren't several blocks appended to a data file at present?

Yes. I know it’s better to avoid using the append mode to improve the performance. But if the data layout is changed, do we need a index file? In the local file or HDFS storage type, the index file will maintain the relation of block offset in the single one file.

I have another question that will we directly use the object store api or to use the hadoop filesystem api to support this?
If it is the former, maybe it’s better to introduce the uniffle dedicated filesystem api to wrap different concrete filesystems, also including the cos.

the index file seems to be not needed anymore.

Emm.. Maybe not. If having no a index structure, it means we don’t a global vision to determine find files we needed in object store, especially for local order. WDYT? @jerqi

Every data file should have index file.

The performance will not be good, as we have to read all index files in one time for doing split segments.

Every data file should have index file.

The performance will not be good, as we have to read all index files in one time for doing split segments.

We can read only one index file.

I have another question that will we directly use the object store api or to use the hadoop filesystem api to support this?
If it is the former, maybe it’s better to introduce the uniffle dedicated filesystem api to wrap different concrete filesystems, also including the cos.

In fact, Can an abstract storage layer be provided in uniffle, and each manufacturer implements the necessary storage interface?
It does not care whether the concrete class uses the native API (such as S3, etc) or the HCFS file system.

Any update on this? @yuyang733 cc @jerqi @advancedxy
If object store is supported, I will use this to store huge partition to reduce HDFS pressure for iQiyi. And this is an important feature for uniffle