horizon reports incorrect fee charges for fail fast transactions

When submitting erroneous transactions, which stellar-core fails instantaneously  (without being submitted for consensus), the resulting XDR (TransactionResult) is shown and has an incorrect feeCharged value. It shows the value of fees offered, although they haven't been charged.
This is confirmed to happen with txBAD_AUTH, txBAD_AUTH_EXTRA, txTOO_EARLY and txINSUFFICIENT_FEE.
It does not happen with txBAD_SEQ and txNO_ACCOUNT. Probably because these have special handling in horizon.
According to @MonsieurNicolas, while these TransactionResult XDRs are coming from stellar-core, they should be presented differently to the clients, as the transactions were never submitted for consensus.

Looking at the different results that Horizon returns...
compare a "fast fail transaction":
{
  "type": "https://stellar.org/horizon-errors/transaction_failed",
  "title": "Transaction Failed",
  "status": 400,
  "detail": "The transaction failed when submitted to the stellar network. The `extras.result_codes` field on this response contains further details.  Descriptions of each code can be found at: https://www.stellar.org/developers/learn/concepts/list-of-operations.html",
  "extras": {
    "envelope_xdr": "AAAAAP4owAbexNelaqr+1bdrDCin1dvMVa4R2NLWuvFYJHYDAAAAZACFZP0AAAACAAAAAAAAAAAAAAABAAAAAAAAAAEAAAAA/ijABt7E16Vqqv7Vt2sMKKfV28xVrhHY0ta68VgkdgMAAAAAAAAAAEgfIoAAAAAAAAAAAlgkdgMAAABAYLPUBFtSxFn4K8iLdIVS9smneYt0s+iPVvyoGNhnCvOyXBZ2pb8xaSpK5Ly8wk242oLMsRwj1kReU9JHyW/DAxRoo20AAABAwEeuMC/sMErk1okvlx2jbn/JoYjR2syKdyWp7kp1reF/Q4J06cpJD+VuTTt2hH6xHHP04Zeauwc86CNDJXKDCQ==",
    "result_codes": {
      "transaction": "tx_bad_auth_extra"
    },
    "result_xdr": "AAAAAAAAAGT////2AAAAAA=="
  }
}

with a "transaction failed during consensus":
{
  "type": "https://stellar.org/horizon-errors/transaction_failed",
  "title": "Transaction Failed",
  "status": 400,
  "detail": "The transaction failed when submitted to the stellar network. The `extras.result_codes` field on this response contains further details.  Descriptions of each code can be found at: https://www.stellar.org/developers/learn/concepts/list-of-operations.html",
  "extras": {
    "envelope_xdr": "AAAAAP4owAbexNelaqr+1bdrDCin1dvMVa4R2NLWuvFYJHYDAAAAZACFZP0AAAADAAAAAAAAAAAAAAABAAAAAAAAAAgAAAAA4d85QWXBamLF7ytpq6s/T6XJ6BEEwxqektkXCBRoo20AAAAAAAAAAVgkdgMAAABA8AEqVQ+aaopkdOj4ML+ppVDkux/24MOvOx9+cufSfmlycszZhktRB/fARS+zIII6Wvqv6FjXxRuuRd0og/FQBg==",
    "result_codes": {
      "transaction": "tx_failed",
      "operations": [
        "op_no_account"
      ]
    },
    "result_xdr": "AAAAAAAAAGT/////AAAAAQAAAAAAAAAI/////gAAAAA="
  }
}

with a "transaction succeeding during consensus":
{
  "_links": {
    "transaction": {
      "href": "https://horizon-testnet.stellar.org/transactions/8023206957eaef27e8c632356ef073abec850d2b2297a889bfae1148517fd001"
    }
  },
  "hash": "8023206957eaef27e8c632356ef073abec850d2b2297a889bfae1148517fd001",
  "ledger": 8742157,
  "envelope_xdr": "AAAAAP4owAbexNelaqr+1bdrDCin1dvMVa4R2NLWuvFYJHYDAAAAZACFZP0AAAABAAAAAAAAAAAAAAABAAAAAAAAAAEAAAAA/ijABt7E16Vqqv7Vt2sMKKfV28xVrhHY0ta68VgkdgMAAAAAAAAAAACYloAAAAAAAAAAAVgkdgMAAABA/5xdjP2LOrDYdCTTuaLztgdA7XF/FxWuZ7+rTLO0zv/E1KtBJJxHxgNSgNjEpX+BPE14kpJCJVH184fj51j+DQ==",
  "result_xdr": "AAAAAAAAAGQAAAAAAAAAAQAAAAAAAAABAAAAAAAAAAA=",
  "result_meta_xdr": "AAAAAAAAAAEAAAAA"
}

I would expect the "transaction failed during consensus" to look pretty much like the last one (maybe keeping the result nested as to avoid the accidental consumption of "result_xdr"). In particular, it should have the ledger information for when it was rejected.
Also, the error message in that case is wrong: it failed during consensus, not when the transaction was submitted.
If we had this information in the result, then distinguishing a "fast fail" from a "real" failure (that most people don't care about), would be trivial.
Maybe something like this (which would be aligned with solving #309 ):
{
  "type": "https://stellar.org/horizon-errors/transaction_failed",
  "title": "Transaction Failed",
  "status": 400,
    "_links": {
    "transaction": {
      "href": "https://horizon-testnet.stellar.org/transactions/8023206957eaef27e8c632356ef073abec850d2b2297a889bfae1148517fd001"
    }
  },
  "hash": "8023206957eaef27e8c632356ef073abec850d2b2297a889bfae1148517fd001",
  "ledger": 8742157,
  "detail": "The transaction failed when processed by the stellar network. The `extras.result_codes` field on this response contains further details.  Descriptions of each code can be found at: https://www.stellar.org/developers/learn/concepts/list-of-operations.html",
  "extras": {
    "envelope_xdr": "AAAAAP4owAbexNelaqr+1bdrDCin1dvMVa4R2NLWuvFYJHYDAAAAZACFZP0AAAADAAAAAAAAAAAAAAABAAAAAAAAAAgAAAAA4d85QWXBamLF7ytpq6s/T6XJ6BEEwxqektkXCBRoo20AAAAAAAAAAVgkdgMAAABA8AEqVQ+aaopkdOj4ML+ppVDkux/24MOvOx9+cufSfmlycszZhktRB/fARS+zIII6Wvqv6FjXxRuuRd0og/FQBg==",
    "result_codes": {
      "transaction": "tx_failed",
      "operations": [
        "op_no_account"
      ]
    },
    "result_xdr": "AAAAAAAAAGT/////AAAAAQAAAAAAAAAI/////gAAAAA=",
    "result_meta_xdr": "AAAAAAAAAAEAAAAA"
  },
}

I'm just going to leave this right here with regards to error response changes from horizon:  https://tools.ietf.org/html/rfc7807

@bartekn do you know if this is still the case?

@bartekn do you know if this is still the case?


Don't execute lines containing only whitespaces

A file with trailing whitespace could lead to something like this:
CONNECT OK
DROP OK, 1 row affected  (0.058 sec)
CREATE OK, 1 row affected  (0.193 sec)
SQLActionException[Failed to parse source [{"stmt": ""}]]

Coverage remained the same at 83.982% when pulling 139f4b9b3d3ea138280926f932f03f0bfd809585 on j/remove-empty-line into 2c24accedb3f95e09861c982b5a06c4a39ad8523 on master.

Codecov Report

Merging #238 into master will not change coverage.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master     #238   +/-   ##
=======================================
  Coverage   83.98%   83.98%
=======================================
  Files          19       19
  Lines        2235     2235
=======================================
  Hits         1877     1877
  Misses        358      358

Impacted Files
Coverage Δ

src/crate/crash/command.py
86.45% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2c24acc...139f4b9. Read the comment docs.

changelog entry?

I think that code was still unreleased and in the released version it appears to have been ignored already


gitter channel ?

Many projects using it already ( scala/scala.js/sbt/scala-ser/....)  , i almost forgot irc! . it will be cool if we have a gitter channel for scalatest too :)

Good idea. Not sure how to create one yet but I think that would be nice to do. We'll try and figure it out then post here once it is working.
Bill

it super simple login to  u r gitter account
-> create Room
-> choose Repository
-> done (gitter will send a automated PR )

We created a gitter channel several months back , but never closed this issue. Closing now.


[Port] Add Product Feedback survey link to welcome page (#14705)

Closes https://github.com/microsoft/azuredatastudio/issues/14738
Links to https://aka.ms/AzureSQLSurvey

This was approved by Ken so once it passes checks I'm planning on merging it in


feat: support displaying AVIF image format

Description
Also support avif image format and show as image instead of file in thumbnail preview. CanIUse shows browser support is high and currently used browsers are covered 88%.

[x ] I have read and understand the CONTRIBUTING.md document in this repository.

Type of change

[ x] Chore (non-breaking change which does not add functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Change to the templates directory (does not affect core functionality)
[ ] Change to the examples directory (does not affect core functionality)
[ ] This change requires a documentation update

Checklist:

[ ] I have added tests that prove my fix is effective or that my feature works
[ ] Existing test suite passes locally with my changes
[ ] I have made corresponding changes to the documentation

Thank you for your PR! I have merged it into the 3.0 alpha branch in 911e902. The package we use for getting the image size in 2.0 throws an error when uploading an avif image, so I won't merge this to main. For 3.0, we switched to a different package which supports avif though, so I'll add it there :)


https://app.clubhouse.io/augur/story/14126/adjust-implementation-of-local-node-storage

Pull Request Test Coverage Report for Build 845

3 of 6 (50.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 65.504%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/modules/app/actions/init-augur.js
3
6
50.0%

Totals

Change from base Build 844:
0.0%

Covered Lines:
3480

Relevant Lines:
4809

💛  - Coveralls


Update wrapper to latest nightly

Signed-off-by: Rodrigo B. de Oliveira<EMAIL_ADDRESS>

@bot-gradle test and merge

Pre-tested commit build failed.


provide a more honest view into the size of the collector queue

Update the queue size Gauge when it gets full, and do the periodic update less often so that it's less likely to overwrite an update from a full queue.

Signed-off-by: Kevin Wojcik<EMAIL_ADDRESS>

Coverage remained the same at 100.0% when pulling cb7c2b5f8a6f01f6c86df22f04a25c514d9313c6 on kwojcik:queuesize into a2ed9b854fa9ef2ce4eb97290d083b73dfbc9bcc on jaegertracing:master.

Hm, I am not a big fan of reporting less often.
I was actually thinking of changing jaeger-lib/metrics to support Histogram instead of Timer. It would allow using Histogram for queue depth.


Optimise requeue all - resolves #244

Only load job ids when doing requeue all, instead of instantiating job objects, to save resources.
Also added gitignore of pycharm project.

Thanks!


Deprecate and remove ML Flow and InnerEye bundles

Remove the ML Flow and InnerEye workspace services due to their deprecation and lack of maintenance. Users can retrieve previous versions if needed. This change addresses issue #4127.

/test-extended

tests fails due mlflow being non-existent so forcing them through

/test-force-approve


dompdf font cache directory

I've noticed that unset font cache produces PHP files in /vendor/dompdf/dompdf/lib/fonts/
Could you set a Laravel-comform location?

I'm not really sure how this affects Cashier? What's the problem exactly?

Thank you for your response.

What's the problem exactly?

.php files are created under /vendor - which is considered read-only after deploy.

Since this seems to be the default behavior for dompdf I don't think this is an actual issue. You can set the font cache with an option to the Dompdf class but since we don't that by default and no one has complained about this so far I don't believe this is a bug.
https://github.com/laravel/cashier-braintree/blob/2.0/src/Invoice.php#L239
Can you perhaps open an issue with them to ask if this default behavior is wanted? For me personally I don't see any issue here.


Upgrade QUnitJS from 1.10.0 to 1.23.1

Hi, @minasmart
Kindly help review the changes here to node-qunit before I create a PR into kof/node-qunit
cc @kof

cc @tessalt

cc @jzaefferer

Why wast test a global before, and now nested under the QUnit global? Is there an assignment missing somewhere?

@minasmart, test is still available actually. I felt being explicit is better.

The PR should have a single purpose. If the purpose is to support qunit 1.20, then do that. If you want it to be more explicit (which is fine), that's a separate PR. One thing is an enhancement, the other is an opinionated choice that the maintainer may or may not be for. 😄

true! 😄 .. updating ...

updated @minasmart. Thanks! 😸

Looks good 👍


Help help!!

Hello, when I dragged the AnimDessin2 to the ExtensionManager, it said that "You need to reinstall Adobe Creative Cloud Desktop app"
I tried, but still can't work:(
Thanks

Have you tried the ZXP installer from anastasiy, main readme
Also its more helpful if you state your OS (Windows/OSX), Photoshop version and possible screenshots or errors

Have you tried the ZXP installer from anastasiy, main readme
Also its more helpful if you state your OS (Windows/OSX), Photoshop version and possible screenshots or errors


Export TestDataGenerator class

I've found myself wanting to reach for this when trying to create benchmarking in an external repo, and potentially useful within web5-js (I think they duplicate much of this class though).
A community member is also looking to create some benchmarking tools and was asking for this to be exported.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (a6f95fb) 98.83% compared to head (d89bde9) 98.83%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #649   +/-   ##
=======================================
  Coverage   98.83%   98.83%
=======================================
  Files          71       71
  Lines        9240     9243    +3
  Branches     1391     1391
=======================================
+ Hits         9132     9135    +3
  Misses        100      100
  Partials        8        8

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@thehenrytsai hopefully not! I just think we need to be careful if/when we decide to use this in web5-js. Maybe when that time comes some refactoring would be done to split some duplicated stuff into their own classes. 🤷


Feature/view store compression properties

Adds two new properties to view description.
primarySortCompression and storedValues

Wrote some text for release notes about covering indexes (correct term?) and conditionOptimization, and added the latter to SEARCH options.


changed to always use 4dn accessions

Coverage remained the same at 51.367% when pulling 1c09c404b1d6f9c15b5c752d6f3c8c90fedd1c01 on test-accessions into 37ec0dc7896442554a7ade4d2d0d11b842421f1f on master.


Language edits and bugfixes

This PR contains the first batch of major changes since the repo was published - conforming to 2 categories: (1) language edits intended to formalize or clarify the text and (2) bug fixes to users' issues.

The Notebooks were originally written to be a bit more conversational than formal, and this lead to some areas with reduced clarity. As such, a batch of edits were made to eliminate figures of speech, idioms, etc.
We address/ Fix #171 and #173 (both on DayNight.ipynb)

#171  focuses on overwrite functionality
#173 ensures that the right type of data (TIME-TAG) is selected by the user before running a function

@larrybradley @eteq
Please also see commit 5ca93ef: which fixes issue #173 by checking that all the files are taken in TIME-TAG mode.
The commit also fixes a couple typos/language changes.

I've updated based on feedback from reviewer @eteq, now merging the PR.


Migration engine should stop hardcoding permissiveMode=true

The migration engine currently has a "permissive mode" which makes it try to keep going if it encounters any exceptions during migration.  This is helpful in allowing us to test out early versions of the migration engine while it is still being implemented, but it has the danger of misleading users into thinking they have a high quality migration when they don't.
Before shipping the migration engine we should either turn off permissive mode or find a way to reassure ourselves that the migrations produced by the tool are very close to accurate even if an exception occurs.

Per @stereotype441 , this no longer fits what we've seen in practice.


Access errors

All files are rejected, due to access errors
Even though, none of these files are actually infected
<IP_ADDRESS> | 7 (Edit) | /var/www/public/assets/Uploads/myimage.jpg | admin/queuedjobs/Symbiote-QueuedJobs-DataObjects-QueuedJobDescriptor/EditForm/field/QueuedJobDescriptor | Deleted, File was infected | Access denied. ERROR
-- | -- | --
<IP_ADDRESS> | 7 (Edit) | /var/www/public/assets/Uploads/myimage.jpg | admin/queuedjobs/Symbiote-QueuedJobs-DataObjects-QueuedJobDescriptor/EditForm/field/QueuedJobDescriptor | Infected, Pending Action | Can't open file or directory ERROR

I'd be curious to know if anyone else is having this issue too? I know we're using this module across a few projects without problems. I assume you've checked your folder permissions.

I can see the problem, after looking at it a bit closer.
The path is /var/www/public. This is incorrect, it needs to be /var/www/{$domain}/web/public
Seems the path is detected wrong in the scan?


[6.0] SILGen: Reabstract subexpr lvalue before ABISafeConversion.

Explanation: Fixes a compiler crash and/or miscompile when accessing a property of function type, in a generic context, from an extension with same-type constraints, in Swift 5 mode with implicit @Sendable conversions introduced.
Scope: Bug fix.
Issue: rdar://130016855
Original PR: https://github.com/swiftlang/swift/pull/75193
Risk: Low. Fixes a relatively rare problem introduced by strict concurrency checking.
Testing: Test case from bug report, Swift CI
Reviewer: TBD

@swift-ci Please test


Add spider/radar chart

Added a visualization: Spider/Radar chart (using D3)
It has options to 'group by' selected columns, and compare the categories on a single metric for all of those columns. The comparisons are shown on the axis of the graph, each axis represents a category.
There is also functionality to set the max value of the graph (outermost circle), this 'stretches' the axes on which the metric's values are plotted.
I have not added any tests, as I am unsure of what I need to test, and how I need to do this - more than happy to add, just need some help there.
Credit to Nadieh Bremer for the original JS and CSS, https://www.visualcinnamon.com
(picture below)

Codecov Report

Merging #4871 into master will decrease coverage by 0.07%.
The diff coverage is 37.5%.

@@            Coverage Diff             @@
##           master    #4871      +/-   ##
==========================================
- Coverage   76.96%   76.88%   -0.08%
==========================================
  Files          44       44
  Lines        8534     8550      +16
==========================================
+ Hits         6568     6574       +6
- Misses       1966     1976      +10

Impacted Files
Coverage Δ

superset/viz.py
79.42% <37.5%> (-0.46%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7193a47...d17a836. Read the comment docs.

Can you add an example in data/__init__.py this will also add test coverage on the python class. Also please rebase.

@mistercrunch
in data/init.py, you mean load sample data and fit the spider as a slice in an example dashboard? Will need a bit of guidance through this first contrib to get comfortable adding other visualizations

@peter-stuart-turner I think he's mean both correct me if I'm wrong @mistercrunch. I would use these PR for guidance on how to add examples.
https://github.com/apache/incubator-superset/pull/4242

@hughhhh @mistercrunch
Hi guys, so I have written tests, as well as added the graph example to the 'miscelaneous' dashboard.
The checks are still failing. It seems to be because of one of the core tests (See below):
I really want to get this workflow cracked, as I suspect I'll be able to contribute a lot over the next few months/years - and will only get better.
Would one of you mind helping me get this request merged? I am available to chat via skype,whatsapp, facebook etc.

@peter-stuart-turner looks like you want to bump the assert statement to 5

@hughhhh
Still failing. I'm not sure this is code I have written:

"ERROR: InvocationError for command '/home/travis/build/apache/incubator-superset/.tox/py36-postgres/bin/nosetests' (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   py36-postgres: commands failed"
Same with sqlite ERROR:   py36-sqlite: commands failed
ERROR:   py27-mysql: commands failed

Any ideas? I really want to become a full time contributor on this project so want to get this process down.

@hughhhh
Still failing. It asks me to resolve conflicts in index.js? I do this (I think) and then I am faced with more errors below.
I'm not sure this is code I have written:
"ERROR: InvocationError for command '/home/travis/build/apache/incubator-superset/.tox/py36-postgres/bin/nosetests' (exited with code 1)
___________________________________ summary ____________________________________
ERROR: py36-postgres: commands failed"
Same with sqlite ERROR: py36-sqlite: commands failed
ERROR: py27-mysql: commands failed
Any ideas? I really want to become a full time contributor on this project so want to get this process down.

Looks like you want to want to change that 4 to a 5 in core_tests.py line 367.
As far as fixing the merge conflict, one way to deal with it if you're confused is to copy the latest version of that file from master and re-write your change, which is just to add an entry for the spider chart in index.js.

@mistercrunch have managed to get them to pass!!:D

@peter-stuart-turner can you update this PR?

Will do it sometime this week or weekend


Add a "default" in Fern Definition

we always put defaults to X in our query params

https://birchsolutions.slack.com/archives/C02AR20BL7P/p1664479131012489

Something that I see often is a message like defaults to true or defaults to 10.

+1

big +1 to be able to specify the default value

For us (Mammoth), we'd love to be able to get defaults in the object you get from the request itself, e.g. defaulting the fields in the pydantic model itself
The specific example we have right now is that we have an object with an optional start and end date, i'd like to be able to access the start and end date with defaulted values if they're not set by the caller


Pinbuilder noimage

Small fix to make loading GeoJSON files not fail when a maki image isn't available.

@stevage you need to rebase this on Cesium master.  Currently it's pulling in all the changes from our terriajs branch as well.

Yeah, thanks.


Unpacking return values?

So I created typescript definitions for MTA:SA (A multiplayer mod for GTA:San Andreas). I'm fairly new to typescript so excuse me if I did something wrong.
This is my function declaration:
declare function getElementPosition(theElement: element): [number, number, number]

This is how I use it:
let [pX, pY, pZ] = getElementPosition(player);

Example from the wiki says:
local x,y,z = getElementPosition( myElegy )

https://wiki.multitheftauto.com/wiki/GetElementPosition
But the transpiler is making this:
local __destr0 = getElementPosition(player)
local pX = __destr0[1]
local pY = __destr0[2]
local pZ = __destr0[3]

which results in the error "attempt to index local '__destr0' (a number value)

Hmm figured as much :/. Thanks for the heads up though. Maybe I'll modify it then, since in Lua there is no native "array" but a table, which is already implemented (In my local copy) with native lua type definitions.

Sounds like a custom decorator for the function declaration would be ideal here.

Added the /** !TupleReturn */ decorator in 04dc05bcbcc647df84160b35125c58b1a144be5e. Add this to your declarations:
/** !TupleReturn */
declare function getElementPosition(theElement: element): [number, number, number]

And call the function as before:
let [pX, pY, pZ] = getElementPosition(player);

This should cause the code to be translated exactly to the example you gave:
local x,y,z = getElementPosition( myElegy )

Let me know if you have any issues!

This does not work for me:
/** !TupleReturn */
function dostuff(value:string) : [string, number] {
return ["string",42];
}
let [a,b] = dostuff("param");
transpiles to:
function dostuff(self, value)
return {"string", 42}
end
a, b = table.unpack(
dostuff(nil, "param")
)
do i need to configure this somehow? Or is the code not correct?

Hi @felixhirt, this is a very old thread, our annotations syntax has changed since then (to follow jsDoc convention). Try replacing /** !TupleReturn */ with /** @tupleReturn */ ;)

thank you very much... guess i should have read the docs instead of an old issue ;-). google brought me here :-(


Adding Globus section

Adding a brief Globus section listing the current endpoints. This will be replaced in the near future with the new Globus setup, perhaps with its own more detailed page.

Preview: https://mit-orcd.github.io/orcd-docs-previews/PR/PR62/filesystems-file-transfer/transferring-files/#globus

@shaohao99 and @kcnakamu, do you want to take a quick look? If this looks good I'll merge it next week.

Also add link to https://docs.globus.org/faq/globus-connect-endpoints/


NAND memory decode error

Here, E is inverted with a gate, and R/W goes to both final NANDs:

That's not correct, it's reversed, R/W should be inverted, as:

Thanks, I didn't see that....I should've paid closer attention.

I have fixed and updated the schematic.


[Bug]: In Audit log the alignment looks off when the email id is too big .

Is there an existing issue for this?

[X] I have searched the existing issues

Description
The alignment looks off when the email id is too big. this happens both in the Users column and also in the Users Dropdown

Steps To Reproduce

Signup with a new email id in ec-2 instance
Now login with demo user
Go to audit logs
Validate the display of email under users column nd also in users Dropdown

Public Sample App
No response
Version
http://ec2-65-2-149-212.ap-south-1.compute.amazonaws.com/applications

The list of emails is going up due to low vertical height of the viewport; that pushes the container upwards.
The email will be truncated due to width of the container.
@vuiets @Debsourabh Need you input.

@Debsourabh what do we have in the Design System for such scenarios?


Does the LogSoftmax works correctly?

test in examples/mnist_classifier.py,
def accuracy(params, batch):
  inputs, targets = batch
  target_class = np.argmax(targets, axis=1)
  x = predict(params, inputs)
  print(x[0])

the final layer is LogSoftmax, but the output seems not correct,
Starting training...
[-2.3113673 -2.6440005 -2.4797316 -1.79847   -1.6207608 -2.931935
 -3.303906  -2.7275395 -2.2099946 -2.1403143]

Can you say a bit more about what might be incorrect here? The sum of the elementwise-exp of those numbers is 1, which is intended.

I mean the sum of those numbers should be 1, not their elementwise-exp's.

sorry, i misunderstood log-softmax, maybe it will be more convenient if providing loss functions.

Got it! I'll add that to my todo list.


Effect of PreciseBN

Does it have any important effect? Thank you!

I'm not sure whether it is important for the performance of SSOD tasks. I followed the default setup used in Detectron2, so it might matter for vanilla object detection tasks.

Got it, thank you :)


[Run] Customize the size of search bar

Description of the new feature / enhancement
Feature to customize the run bar because it sometimes use majority of the screen area
Scenario when this would be used?
It covers most of the area of screen which is a bit annoying !
Supporting information
No response

Related to #12182

/dup #12654


highlight expressions on hover

The current default behavior is to show subtle highlighting of identifiers on hover. However, this doesn't really mesh well with the position info which is supposed to indicate whether an expression is in a tail (call) position.
What happens is that when you hover over the head term of an application, it will report position: normal, which is fine, but a little bit surprising. You only see the position: tail if you hover slightly in front of the head term but now you don't see anything highlighted in the buffer because there is no identifier under the cursor. What should happen is the entire application should be highlighted.
tldr: the highlighting extents on hover should be calculated by merlin from the shape of the expression at the cursor position rather than being calculated by the editor based on identifiers.

Related to #60.


Python analyser

Hi
I am trying to run dependency check against python and it doesnt seem to do anything. I have seen this is still in experimental phase, but i am not sure if something has been released yet?
In any case, how is this analyser working? I have seen a python package called safety which seems to work well  -- https://pypi.python.org/pypi/safety. Maybe the easiest solution is to integrate this tool in DC?
Thanks

how is this analyser working?

Pretty badly. You need to point it at your site-packages, and then even the HIGHEST confidence CPEs are still wrong.
The safety project uses its own curated database and as far as I can tell doesn't use CPEs or CVEs at all. It simply parses a requirements.txt (or pip freeze) or using pkg_resources.working_set to get exact package names and versions. It might be difficult to express its results in dependency-check's format.

I am trying to run dependency check against python and it doesnt seem to do anything.

How did you try running it?
the cli command dependency-check -s ./ produces an empty report.
I also came across this python package, but I am not sure how to run it. It currently gives me [ERROR] No documents exist even if I try to point to the right directory.
@javixeneize @jeremylong

For me it’s easier and more reliable to run safety, that’s why I suggested
to add this feature in dependency check
El El lun, 2 dic 2019 a las 18:22, Nasif Imtiaz<EMAIL_ADDRESS>escribió:

I am trying to run dependency check against python and it doesnt seem to
do anything.
How did you try running it?
the cli command dependency-check -s ./ produces an empty report.
I also came across this python package
https://pypi.org/project/dependency-check/, but I am not sure how to
run it. It currently gives me [ERROR] No documents exist even if I try to
point to the right directory.
@javixeneize https://github.com/javixeneize @jeremylong
https://github.com/jeremylong
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jeremylong/DependencyCheck/issues/1435?email_source=notifications&email_token=ADBLTZS5AOEH7EQ276H66R3QWVG6RA5CNFSM4FPBHDXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFUNI2A#issuecomment-560518248,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADBLTZSA7PIJHUZLAQ7GAETQWVG6RANCNFSM4FPBHDXA
.

Just a reference, here is example JSON output from safety. Some sort of integration similar to the npm audit analyser would be a nice to have, in order to get reports consistent with dependency-check,
[
    [
        "pyyaml",
        "<4",
        "3.13",
        "Pyyaml before 4 uses ``yaml.load`` which has been assigned CVE-2017-18342.",
        "36333"
    ],
    [
        "pyyaml",
        "<5.3.1",
        "3.13",
        "A vulnerability was discovered in the PyYAML library in versions before 5.3.1, where it is susceptible to arbitrary code execution when it processes untrusted YAML files through the full_load method or with the FullLoader l
oader. Applications that use the library to process untrusted input may be vulnerable to this flaw. An attacker could use this flaw to execute arbitrary code on the system by abusing the python/object/new constructor. See: CVE-2020-
1747.",
        "38100"
    ]
]


Fixed #27656 -- Fix comment verb style according to PEP 257.

Work in progress

@timgraham I'll be making the changes you suggested as well as fixes for the verb style. Also, I noticed that this coding style is not being followed as well. I should probably be fixing that as well - I go through the all of the comments anyway.

In test docstrings, state the expected behavior that each test demonstrates. Don’t include preambles such as “Tests that” or “Ensures that”.

And I assume it is a good idea to also make sure these PEP rules are being followed as well:

Triple quotes are used even though the string fits on one line. This makes it easy to later expand it.
The closing quotes are on the same line as the opening quotes. This looks better for one-liners.
There's no blank line either before or after the docstring. (except for classes)
The docstring is a phrase ending in a period.

Keep this patch focused on django/ only -- don't worry about test docstrings, there's a separate ticket for that.
The PEP rules you mentioned are fine.

Ok. Also I noticed that the wrapping is sometimes different. It can be 72 or less than 79. Since you mentioned it in the commit we should probably make it consistent to 79.

Documentation, comments, and docstrings should be wrapped at 79 characters, even though PEP 8 suggests 72.

If you're making significant changes to the docstring, feel free to adjust the length to 79, otherwise don't make needless changes.

I made some changes. Please see if I am on the right track.

Please do only verb changes in this PR. I think all other changes could be refactored automatically with some tool if we decide to do that. Otherwise, this is just creating a huge unreviewable patch.

So I assume this means don't follow your previous suggestions and do only verb changes. I caught few typos - I will include them as well.

Yes, sorry I had second thoughts about this when I started seeing how many more changes would be required as well as the thought that other changes could be automated.

On further thought, I'd hold off on doing further changes for a couple weeks so that you don't create merge conflicts with all that code that will be removed in Django 2.0 as part of the ending deprecations (https://github.com/django/django/pull/7773). I'll leave a note when it's a good time to continue. Thanks!

I guess we can continue with this ticket since #7773 got merged.

Sure, feel free to rebase. Maybe you can submit some smaller commits for easier review, such as only doing "django.contrib" so that the patch doesn't get so long that GitHub starts displaying "Load diff" buttons. It gets tedious to click those to review things.

I completed all of the changes.

What was your methodology for identifying places that need to be fixed? Working my way through contrib.admin (so don't bother reviewing that one) I see a lot of places where, for example, the first verb in the sentence was fixed but not the second (e.g. "Remove the authenticated user's ID from the request and flushes their". Even grepping for "Returns" shows places that weren't updated. Another phrasing to check for is something like "If no user is retrieved an instance of AnonymousUser is returned." which should be "If no user is retrieved, return an instance of AnonymousUser."  I know this is really tedious, but reviewing a partially completed attempt at this isn't fun either.

django.contrib commit merged in 5411821e3b8d1427ee63a5914aed1088c04cc1ed (with many edits).


Added the ability to use eternal plugins with forma

Added the ability to use eternal plugins with forma.
Also added platform dependency type.

#35

#35


Add Project README

The @effection/contrib package is designed to be a warehouse of curated, emerging helpers for dealing with very common tasks encountered while using Effection.
The goal is to be both a useful repository of modules that we can point to whenever someone asks a question that can solve there problem, but also serve as an educational tool.
Once we are confident that a module in contrib really does represent the "right" way to do things, then the module can graduate to it's own package, or even effection core (in the case where it works with a standard API like workers or websockets)
We need a README that will

Explain the reason @effection/contrib exists
Provide an index to all of the individual modules with a brief explainer on what problem they solve.
describe the best way to use contrib modules and what to expect from them and their place in the ecosystem (emerging techniques) without scaring them away.
Invite contribution

Provide an index to all of the individual modules with a brief explainer on what problem they solve.

Do we want this to be manually entered for each module?

I think we should start by manually entering it for each module, but eventually, and sooner rather than later, we would parse the markdowns Of each individual module and use some structure there to compile an index in the main read me during each release process


Update to<EMAIL_ADDRESS>

Blog pages may need to change to .mdx, .md files are now under a legacy flag

done via #20


HOTFIX: fix subscription narrowing problem

I’m missing an example of what the resolver implementation would look like. Is this requiring folks to include the kind property in their implementation?

Yes, you will need to add kind for "SubscribeAndResolve"

That doesn’t seem helpful. At that point it’s more intuitive to export two different type definitions.

That doesn’t seem helpful. At that point it’s more intuitive to export two different type definitions.

That was my initial idea, but @freiksenet said that we would need to refactor everything then.

Can you elaborate on the difference in effort needed to add kind vs using a different type? Both can be done relatively the same way with a codemod afaik.


gz_TEST: one more ign -> gz

🦟 Bug fix
Fixes failing test
Summary
There is a portion of the UNIT_gz_TEST that is still calling the ign command, but that was removed from gz-tools2, so the test is failing. Just call gz instead.

 https://build.osrfoundation.org/job/ignition_msgs-ci-gz-msgs9-focal-amd64/3/testReport/(root)/CmdLine/MsgHelpVsCompletionFlags/

tools/gz_TEST.cc:120
Expected: (std::string::npos) != (helpOutput.find(flag)), actual:<PHONE_NUMBER>3709551615 vs<PHONE_NUMBER>3709551615
sh: 1: ign: not found

Checklist

[X] Signed all commits for DCO
[ ] Added tests
[ ] Updated documentation (as needed)
[ ] Updated migration guide (as needed)
[ ] Consider updating Python bindings (if the library has them)
[X] codecheck passed (See contributing)
[X] All tests passed (See test coverage)
[ ] While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

I've also removed the include/ignition/msgs/test_config.h redirection header per #302 and a discussion with @methylDragon, which should fix the abi-checker build

ABI checker is still failing with the same issue. It might be from a dirty workspace, I'm going to clear the workspace and rerun the job.

ABI checker is still failing with the same issue. It might be from a dirty workspace, I'm going to clear the workspace and rerun the job.

I think the ABI checking is failing on the target branch; I think the job won't succeed until this has been merged to gz-msgs9

ABI checker is still failing with the same issue. It might be from a dirty workspace, I'm going to clear the workspace and rerun the job.

I think the ABI checking is failing on the target branch; I think the job won't succeed until this has been merged to gz-msgs9

I merged https://github.com/gazebosim/gz-msgs/pull/301/commits/104ea78892cb639d82dbdc60230ac40454d7f244 directly to gz-msgs9 in #303, so I expect CI to be fixed now

I merged 104ea78 directly to gz-msgs9 in #303, so I expect CI to be fixed now

and it's fixed!


Could not find generator 'langchainrb_rails:pgvector' if you follow instructions

HI, I created a new rails installation and got this.
Both with gem install and bundle install, it doesnt find the generator.
$ rails generate langchainrb_rails:pgvector --model=Product --llm=openai
Could not find generator 'langchainrb_rails:pgvector'. (Rails::Command::Base::CorrectableError)
Run `bin/rails generate --help` for more options.

Maybe there are some prerequisite to the created rails repo? I've used

ruby 3.2.1
rails 7.0.8

App is available/reproable here: https://github.com/palladius/langchainrb_rails_test

@palladius It works with my app on rails <IP_ADDRESS> / ruby 3.1.2. I'll have to dive in a bit deeper. Odd.

@palladius I just realized that your Gemfile is missing langchainrb_rails gem. I think if you add it, bundle install, it should just work!


mo.ui.table does not render in HTML export on Windows Chrome/Edge

Describe the bug
Bug report from a user (cc @VivaldoMendes)
Data frames not rendered in the HTML version. I tested it on two different computers (Windows 10 and Windows 11). The data frames are displayed correctly in Firefox but not in Chrome and Edge. I am using marimo 0.7.11, and Python 3.12.4.
Loading time in Chrome and Edge. The fact that the data frames are not displayed in Chrome and Edge (in the static version) may explain why these browsers take a long time to load the entire HTML file (static notebook with 1257KB): around 10/15 seconds. Possibly, they took the time to try to render the data frames, but eventually, they failed and opened the document with the errors. In Firefox, the file opens immediately, and I have no problem with it.
Environment
0.7.11
Windows
Chrome, Edge
Code to reproduce
No response

I have checked this problem in Linux, and it does not exist.
So, it seems that Edge and Chrome's inability to render data frames in the static version is an exclusive problem of Windows.

If anyone has the same problem:

It seems there is a problem in the virtual file system code. Chrome looks for actual files and those are blocked by CORS policy.
using "marimo export html ..." instead of the UI works
This snippet from Claude fixed the problem on my notebooks.  ( I used it before finding out that the CLI version works, maybe it helps the developers fix the bug):

``

If anyone has the same problem:

It seems there is a problem in the virtual file system code. Chrome looks for actual files and those are blocked by CORS policy.
using "marimo export html ..." instead of the UI works
This snippet from Claude fixed the problem on my notebooks.  ( I used it before finding out that the CLI version works, maybe it helps the developers fix the bug):

<script>
  const originalFetch = window.fetch;
  window.fetch = async function(input, init) {
    const url = input instanceof Request ? input.url : input.toString();

    // Check if this is a virtual file request
    if (url.includes('@file/')) {
      // Extract the file name from the URL
      const fileName = url.split('@file/')[1];
      const fullPath = '/@file/' + fileName;

      // Get the base64 data
      const dataUrl = window.MARIMO_STATIC.files[fullPath];
      if (dataUrl) {
        // Convert data URL to blob
        const response = await fetch(dataUrl);
        const blob = await response.blob();
        return new Response(blob, {
          headers: {
            'Content-Type': 'text/csv'
          }
        });
      }
    }

    return originalFetch(input, init);
  };
</script>

I can confirm that using Marimo export on CLI fixes it on my machine.

Any update on this? Would fixing this just a matter of including that JS snippet (or similar) in whatever template marimo is using to render the notebook?
CLI export works, but

requires running a (potentially-long-running) notebook again
#3233

@gabrielgrant we already do this: https://github.com/marimo-team/marimo/blob/main/frontend/src/core/static/files.ts#L11-L41
I missed this snippet that fixes it. I think the fix is the 'Content-Type' which the snippet includes but the original code did not.

@mscolnick Awesome, thanks for addressing! Would test it out, except... #3235 🙃


DAPT codegen data curation tutorial

Description

Added a tutorial to curate data for DAPT codegen use-case with Nemo Curator
Usage

codegen_DAPT\llm_data_collection\notebooks\data_curation-nemo-curator_DAPT.ipynb

Checklist

I am familiar with the Contributing Guide.
New or Existing tests cover these changes.
The documentation is up to date with these changes.

@jvamaraju Thank you for your contribution. Here's some initial feedback after spending some time going through the PR.
High level comments
Considering that this is a tutorial, we need to have explicit entry points for each topic where we are attempting to teach users how to accomplish certain things, or show them the best practices. We can follow a structure roughly similar to what you walked us through in the weekly meetings. Consider these (hypothetical) steps. I think the tutorial greatly benefits from having a "flow" where users can experience each data curation step by going through the flow:

Step 1: Dependency Installation

Before users do anything, they need to understand all the code dependencies they need to fulfill. This includes pip package installs, PDF cleaner installation, etc. We could either have a notebook for this or a README file. I think a notebook would be better due to its interactive nature.

Step 2: Data Acquisition

Substep 1: Users need to download the data from the Google Drive share. Show them where the data is located, and what directory this needs to be under.
Substep 2: Users need to run some shell scripts to traverse the downloaded data and create the relevant meta data.
... [The rest follows] ...

Step 3: Data Curation using NeMo Curator

Substep 1: Install dependencies
Substep 2: .....

Step 4: Data Curation without NeMo Curator

Substep 1: .....
Substep 2: .....

Specifically, I think the data acquisition step also deserves a dedicated notebook where users can walk through the sequence of operations they need to do before they can follow along with the rest of the tutorial. Also, if there are shell scripts that need execution, we can make cells for them inside the notebook using % operator (see this example). This way, users won't have to worry about the order of executing different scripts.
Lastly, where relevant, I suggest clearing the cell outputs from the notebooks. I found myself scrolling a lot while going through the code and understanding the structure. For instance, this notebook, is around 2MB and most cell outputs contain logs that users would be able to reproduce if they ran it locally. We should be safe to remove all those outputs from the checked in file. In some cases, it might make sense to keep that output (e.g. when we're trying to show a sample output to the user), so it's fine to keep them.
Organization
The tutorial consists of 4 main types of files:

Python scripts
Shell scripts
Notebook files
Data files consisting of

Images (e.g. PNG)
JSONL (e.g. this one)
Data that the user downloads.

One challenge I faced while going through the repo was that different files appear in different places. In order to greatly enhance the readability and organization, I suggest the following structure (this is a rough sketch, feel free to incorporate based on your intuition of course!) :

Aggregate all scripts inside one folder (e.g. scripts). It can have subfolders to organize based on the step of data curation (e.g. scripts/1-data-acquisition or scripts/2-data-curation).
Organize Python scripts into logical units. For instance, if there are scripts that are helpers, they can all go inside a helpers directory. Reusable code can go into a common folder for instance.
Aggregate all images in the same place. This has been already done, but some PNG files are still floating around.
Create a centralized place for all other types of data. Perhaps the dataset folder that you have can serve as this? I'm thinking we could create a specific folder inside it dataset/downloaded where users can download the dataset under. Then everything else can reside in dataset directly (e.g. JSONL files).

PDF Cleaner
This is a very useful tool and is standalone. I think it can be reused over and over throughout the repo. In its current form, it looks like it is a step of the tutorial. I'll defer to @ayushdg or others to comment on the best organization for this. It also contains docker image creation, etc. which appear to be out of the scope of the tutorial. Let's brainstorm on the most logical way of integrating this tool into the repo.

@Maghoumi  I have incorporated all the feedback. Specifically
Added license headers to all the files.
Cleaned up and unify function names
Added docstrings to all functions
Thoroughly tested with all the sources (currently it limits to 10 sources of Wikipedia, GitHub and arXiv).
Please review.

@Maghoumi  Please look at my comments and updated repo.

[like]  Janaki Vamaraju reacted to your message:

From: Mehran Maghoumi @.>
Sent: Thursday, July 11, 2024 6:15:41 PM
To: NVIDIA/NeMo-Curator @.>
Cc: Janaki Vamaraju @.>; Mention @.>
Subject: Re: [NVIDIA/NeMo-Curator] DAPT codegen data curation tutorial (PR #88)
@Maghoumi commented on this pull request.

On tutorials/distributed_data_classification/distributed_data_classification.ipynbhttps://github.com/NVIDIA/NeMo-Curator/pull/88#discussion_r1674457723:
These files are still showing as changed. Here's how I would resolve it:
git branch backup  # Make a backup of your branch as-is in case something gets messed up.
git reset 224800b4f141b41cdea376daf3c55a0cd788df47~1  # Undo your entire commit history, but keeps changed files.
git add tutorials/dapt-curation
git commit -sS -m "Your commit message"
git reset --hard  # to get rid of irrelevant changes
git push -f
Basically the idea is to only add the stuff under tutorials/dapt-curation to your branch, then commit everything as a new commit. Let me know if this makes sense.

In tutorials/dapt-curation/README.mdhttps://github.com/NVIDIA/NeMo-Curator/pull/88#discussion_r1674440723:

@@ -0,0 +1,67 @@
+# Data curation for DAPT (Domain Adaptive Pre-Training)

+ChipNeMo is a chip design domain adapted LLM. LLama models are continually pre-trained with 20B plus tokens on domain-specific chip design data, including code, documents, etc., based on NeMo foundation models and then fine-tuned with instruction datasets from design data as well as external sources.
+
+Here, we share a tutotial with best practices on data curation for DAPT (domain-adaptive pre-training) for a ChipNeMo-like code generation use case.
+
+* In this tutorial, we will leverage chip domain/hardware datasets from open-source GitHub repositories (.code/sources/github_repos.jsonl), wiki URLs (.code/sources/wikipedia_urls.jsonl), and academic papers(./sources/arxiv_urls.jsonl).
+
+* \code\data contains curated data after processing
+
+* Notebook for data curation with NeMo Curator: \code\DAPT_data_curation.ipynb
Yes. I suggest keeping a copy of it in a branch or something for testing.
Given that this repo is production-grade, we're not allowed to push experimental/debugging content to it.

In tutorials/dapt-curation/README.mdhttps://github.com/NVIDIA/NeMo-Curator/pull/88#discussion_r1674442399:

@@ -0,0 +1,67 @@
+# Data curation for DAPT (Domain Adaptive Pre-Training)

+ChipNeMo is a chip design domain adapted LLM. LLama models are continually pre-trained with 20B plus tokens on domain-specific chip design data, including code, documents, etc., based on NeMo foundation models and then fine-tuned with instruction datasets from design data as well as external sources.
+
+Here, we share a tutotial with best practices on data curation for DAPT (domain-adaptive pre-training) for a ChipNeMo-like code generation use case.
+
+* In this tutorial, we will leverage chip domain/hardware datasets from open-source GitHub repositories (.code/sources/github_repos.jsonl), wiki URLs (.code/sources/wikipedia_urls.jsonl), and academic papers(./sources/arxiv_urls.jsonl).
+
+* \code\data contains curated data after processing
+
+* Notebook for data curation with NeMo Curator: \code\DAPT_data_curation.ipynb
+
+The small size of this dataset makes it ideal for creating and validating data curation pipelines on a local machine or a cluster.
Since the goal was to reproduce ChipNeMo's data curation, shouldn't we still include all the sources?
We can use the limit=10 for the downloaders (along with a note for users) to maintain the ability of running the whole thing on a local machine.

In tutorials/dapt-curation/code/sources/github_repos.jsonlhttps://github.com/NVIDIA/NeMo-Curator/pull/88#discussion_r1674446964:

@@ -0,0 +1,6 @@
+"google/skywater-pdk-libs-sky130_fd_sc_lp"

I thought the legal issue was with bundling all the data into the Google Drive share. Are you saying we can't include a link to the repositories, Wikipedia pages and arXiv papers?

In tutorials/dapt-curation/code/utils.pyhttps://github.com/NVIDIA/NeMo-Curator/pull/88#discussion_r1674448258:

Returns:

   DocumentDataset: The cleaned and unified dataset.

"""
cleaners = Sequential(

   [

       # Unify all the quotation marks

       Modify(QuotationUnifier(), text_field="text"),

       # Unify all unicode

       Modify(UnicodeReformatter(), text_field="text"),

   ]

)
return cleaners(dataset)

+def filter_txt(dataset: DocumentDataset) -> DocumentDataset:
Yes, but let's rename to filter_text.

In tutorials/dapt-curation/code/utils.pyhttps://github.com/NVIDIA/NeMo-Curator/pull/88#discussion_r1674449907:

   print(doc)

   if "text" in doc:

       # print("initial", doc["text"])

       # Extract the comment content between /* and */

       start_idx = doc["text"].find("/*")

       end_idx = doc["text"].find("*/")

       if start_idx != -1 and end_idx != -1:

           comment = doc["text"][start_idx : end_idx + 2]

           # Create a Dask DataFrame from the given data

           comment_df = dd.from_pandas(

               pd.DataFrame([{"text": comment}]), npartitions=1

           )

           # Create a DocumentDataset instance

           comment_doc_dataset = DocumentDataset(comment_df)

Adding @ryantwolfhttps://github.com/ryantwolf and @ayushdghttps://github.com/ayushdg to comment on how to workaround this issue.

In tutorials/dapt-curation/code/utils.pyhttps://github.com/NVIDIA/NeMo-Curator/pull/88#discussion_r1674459060:

"""

Redacts personally identifiable information (PII) from a given dataset.

Args:

   dataset (DocumentDataset): The dataset containing documents with PII.

Returns:

   DocumentDataset: The redacted dataset with PII replaced by a generic value.

"""
DEFAULT_MAX_DOC_SIZE = 3000000
redactor = Modify(

   PiiModifier(

       supported_entities=[

           "PERSON",

           "EMAIL_ADDRESS",

           #                                "URL",

Thanks.
The usage is already documented and shown in other tutorials.

In tutorials/dapt-curation/code/utils.pyhttps://github.com/NVIDIA/NeMo-Curator/pull/88#discussion_r1674450814:

Returns:

   DocumentDataset: The redacted dataset with PII replaced by a generic value.

"""
DEFAULT_MAX_DOC_SIZE = 3000000
redactor = Modify(

   PiiModifier(

       supported_entities=[

           "PERSON",

           "EMAIL_ADDRESS",

           #                                "URL",

           #                                "PHONE_NUMBER",

           #                                "ADDRESS",

           #                                "LOCATION"

       ],

       anonymize_action="replace",

       #             batch_size=1000,

Then we should add if .... else.... block. Leaving commented out code is not allowed in production grade code.

In tutorials/dapt-curation/code/main.pyhttps://github.com/NVIDIA/NeMo-Curator/pull/88#discussion_r1674443239:

+from nemo_curator.modifiers.pii_modifier import PiiModifier
+from nemo_curator.modifiers.unicode_reformatter import UnicodeReformatter
+from nemo_curator.modules import ExactDuplicates
+from nemo_curator.modules.modify import Modify
+from nemo_curator.utils.distributed_utils import get_client
+from nemo_curator.utils.file_utils import (

get_all_files_paths_under,
separate_by_metadata,
+)
+from nemo_curator.utils.script_utils import add_distributed_args, parse_client_args

+SCRIPT_DIR_PATH = os.path.dirname(os.path.abspath(file))
+DATA_DIR = os.path.join(SCRIPT_DIR_PATH, "data")
+
+#
+# TODO
Did you also remove the TODO notes here? If already removed, it's not showing up here for some reason.
—
Reply to this email directly, view it on GitHubhttps://github.com/NVIDIA/NeMo-Curator/pull/88#discussion_r1674457723, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BIZL5HQL7OOUALHWEXGZI5LZL3DU3AVCNFSM6AAAAABIOERV7GVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDCNZSGYZDCMRYHE.
You are receiving this because you were mentioned.Message ID: @.***>

pre-commit.ci autofix

pre-commit.ci autofix

Closing in favor of #160


Release testing 1.3.2

Is your enhancement related to a problem? Please describe
follow-up https://github.com/containers/podman-desktop-extension-ai-lab/issues/2027
Describe the solution you'd like

We will run a smoke test for AI Lab 1.3.2
Manual testing for ibm/merlinite-7b-GGUF
Manual testing for lmstudio-community/granite-3.0-8b-instruct-GGUF
Main feature was model import

Describe alternatives you've considered
No response
Additional context
No response

QE has finished testing of AI Lab 1.3.2
Following issues were found:

Imported model details page is empty
Granite 3.0 gets into a loop when generating answer using Chat Bot app as backend
Application takes too long to restart (MacOS)
Playground - model does not respond correctly to system prompt, possibly malformed (?), need to update testing sheet

Testing sheet: https://docs.google.com/spreadsheets/d/1TuJWb7c1tSbh-4ZPoYESAviDZiEhS15CmGb_Ky_Z1Cw/edit?gid=686224322#gid=686224322
QE approves this release.


[BUG] version number is incompatible with existing ES clients

Describe the bug
The API which provides version includes a field version.number. Clients parse this value to determine what version of ElasticSearch is running, and is used for checking API compatibility.
In ElasticSearch 7.10, this value is reported as 7.10.
In OpenSearch 1.0.0, the value is 1.0.0-SNAPSHOT.
Existing clients which parse this string are unable to communicate with OpenSearch when they rely on the value of the version.
To Reproduce
Steps to reproduce the behavior:

Start ElasticSearch 7.10
curl http://localhost:9200
Read version.number.
Start OpenSearch
curl http://localhost:9200
Read version.number.

Expected behavior
Because OpenSearch uses the same API as ElasticSearch 7.10.2, the value of version.number should be 7.10.2.
Expected:
  "version" : {
    "number" : "7.10.2",

Actual:
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.0.0-SNAPSHOT",

Plugins
N/A
Screenshots
N/A
Host/Environment (please complete the following information):
N/A
Additional context
We confirmed that this is the cause of #667. It also appears that this may be causing #669.

Since the problem is only with the number property, a possible solution would be:

Retain the number property as matching the ElasticSearch API version
Add a new property representing the OpenSearch version.

For example:
"version" : {
    "distribution" : "opensearch",
    "number" : "7.10",               // This is the ES API
    "opensearchVersion" : "1.0.0"    // This is the OpenSearch version
    ...
}

There is probably a property name to use rather than "opensearchVersion." But, the advantage of this solution is that callers can know the ElasticSearch API version and the actual OpenSearch version (if that matters to the client).

Doesn't feel right that we're retaining ES number anywhere. Are we saying that we don't want clients to have to modify anything to work against OpenSearch?

Yes, I believe that clients should not have to make changes to migrate from ElasticSearch 7.10.2 to OpenSearch 1.0. We've already found that Logstash with the Logstash ES plugin can work with OpenSearch if we manually change version.number to 7.10.2.
The introduction blog post also indicates this is the case.

The Amazon OpenSearch Service APIs will be backward compatible with the existing service APIs to eliminate any need for customers to update their current client code or applications.

Clients could assert compatibility with OpenSearch by using the new distribution field which has already been added.
Additionally, the solution I proposed above may serve as a stepping stone from OpenSearch 1.x to 2.x. The change allows existing clients to work with OpenSearch 1.x without modifications (assuming other APIs are still compatible). Clients and libraries could eventually move to support OpenSearch only, or attempt to support both ElasticSearch and OpenSearch by utilizing the distribution field. Perhaps OpenSearch 2.x would remove the version.number entirely in favor of version.opensearchVersion.
For example, clients could start to write code along these lines. It would be compatible with the suggestions above and would continue to work if version.number were removed in 2.x.
if version.distribution == 'opensearch':
  if version.opensearch.startsWith('1.x'):
    handleOpensearch1();
else:
  if version.opensearch.startsWith('7.x'):
    handleElasticsearch7();

I am 💯  with you on backward compatibility, but OpenSearch didn't break backward compatibility by changing the value of version.number (the API continues to return the same fields, including number, but a value has changed to 1.0-...). Even though reverting it to 7.x seems to work around the problem, someone should figure out what the actual root cause of #667 is without that. The exception produced (java.lang.IllegalArgumentException: mapper [log.level] cannot be changed from type [keyword] to [text]) looks odd to me.
Someone here will definitely get to #667 in order, but don't let them beat you to it :)

@dlvenable raises a great point that this is almost certainly going to be more than a logstash issue: a lot of clients that interact with OpenSearch are going to check the version # to know the wire protocol to use. The proposed solution seems really elegant: keep version.number at 7.10.2 which reflects the API compatibility and lets current clients work without changing, and have a separate openSearchVersion identifier for new clients.

@jcgraybill I think I agree, but I do worry that this could lead to users having broken software more often as a production surprise. We'd be asserting that OpenSearch 1.0 works just like ES 7.10 in every way rather than the fact that it has a backward-compatible API that minimizes client code changes.
Either way, we need to contribute integration tests against OpenSearch to https://github.com/logstash-plugins/logstash-output-elasticsearch. It currently runs against multiple versions of ES.

@dblock , Thanks for engaging on this.
I'm unsure exactly what issues you might be imagining. Are they problems with automated tools expecting specific behavior from specific versions? Or more of a human-related issue?
With the latter, I could see an issue with logging. Code like log.info("ElasticSearch version {}", version.number) would log out ElasticSearch version 7.9.2. That could lead to confusion for human operators.
It is likely a fair assumption that automated clients which parse the version mostly care about the major version, and maybe sometimes care about the minor version. The patch version is probably ignored.
Perhaps OpenSearch could have an alternate patch value? Some possible ideas are:

Format the OpenSearch version into the patch value. Something like MMmmpp (M=major, m=minor, p=patch).
Start at some large version and auto-increment from there
To support a 2-digit only patch, start at a smaller patch version and auto-increment from there

OpenSearch Version
Formatted
Auto-Increment
Auto-Increment w/ 2 Digits

1.0.0
7.2.10000
7.2.1001
7.2.51

1.0.1
7.2.10001
7.2.1002
7.2.52

1.0.2
7.2.10002
7.2.1003
7.2.53

1.1.0
7.2.10100
7.2.1004
7.2.54

2.0.0
8.0.20000
8.0.1000
8.0.51

Now that log line about could read ElasticSearch version 7.9.10000.

Let's be careful not to engineer a ton of cosmetic surgery just to treat a secondary symptom. Doing so risks causing other problems. First things first, the backwards compatibility tests are still disabled; so we absolutely need to re-enable those in order to catch any/all other areas affected by the version rebase. (/cc @harold-wang) That's being worked now.
As for this particular symptom, the root cause is similar to the "opensearch node fails to join" issue - /cc @shwetathareja. In short, since the server side transport layer is "version aware" of the client we can detect what version of the client is attempting to connect and "mask" the OpenSearch version to predecessor 7.10.2 appropriately. This is why we've been super thoughtful to have the 1.0.0 release not introduce any breaking changes from 7.10.2.

Perhaps OpenSearch could have an alternate patch value?

There is already a patch value in the semantic versioning system. (e.g., 1.0.0 is the 0 patch version of Major 1, Minor 0 release).
Are you suggesting a "feature" or "nightly" number? This has been brought up before (in the context of plugins) and I think it's worth discussing as a separate topic independent of this particular bug because I think it has merit. Would you like to open a separate discuss issue?

@nknize , The issue you referenced for "OpenSearch node fails to join" mentions a "backward compatibility mode" solution. Is the idea that OpenSearch would report version.number as 7.10.2 when that flag is set?
Regarding the patch idea, this was an extension of my original proposal. Rather than returning 7.9.2 as the version, we could have a different patch value so that human-operators can clearly see that it is OpenSearch. I offered it as a suggestion to @dblock 's concern:

I do worry that this could lead to users having broken software more often as a production surprise.

To elaborate on the purpose of this bug, we found that Logstash and Filebeats both fail to load templates when running against OpenSearch 1.0.0.

In local experiments, we modified MainResponse to return a hardcoded value:
.field("number", "7.10.2")

After this change, Logstash and Filebeats can run against OpenSearch.
It does appear that this is the cause of #706 as well. I did not create the exact setup that author included. However, both of us saw this same error in OpenSearch 1.0.0.
couldn't load template: 400 Bad Request: {"error":{"root_cause":[{"type":"mapper_parsing_exception"

Again, that manual change to MainResponse was sufficient to get past the error.

@dlvenable,
Just chiming in to say thanks,  and this worked in our testing.  https://github.com/opensearch-project/OpenSearch/issues/706#issuecomment-841582430

Is the idea that OpenSearch would report version.number as 7.10.2 when that flag is set?

No. I'm saying use the versioning system as it is intended (for backward compatibility) and fix MainResponse to return the predecessor version for LegacyClients which wasn't covered in the initial patch that adds the distribution key-value to the MainResponse to identify OpenSearch builds from other forks.

In local experiments, we modified MainResponse to return a hardcoded value:

No hardcoded values. It should be based on the client version. But as I mentioned, I wouldn't treat this symptom just yet while Backwards Compatibility testing is disabled. This could be "patching" one issue and causing another. This "fix" I'm suggesting is on hold until we can smoke test w/ bwc tests enabled.

@harold-wang or @nknize, are you working on this?

@harold-wang or @nknize, are you working on this?

First PR opened

@nknize , As I understand PR #708, this will support the internal node-to-node communication inside an ElasticSearch cluster.
I created this ticket mostly to support external clients. Also, #667 and #706 relate to Logstash and Filebeats. How would ElasticSearch determine the client version in order to change the reported version number?

@nknize , As I understand PR #708, this will support the internal node-to-node communication inside an ElasticSearch cluster.

It will support compatibility with transport layer clients. Those that don't connect via the transport layer and solely rely on a raw http call to obtain server version number in this manner will get the 1.0.0 OpenSearch version; and that should be the correct behavior. Hard coding that to a spoofed version is not the solution, it's a hack, and OpenSearch doesn't provide any forward compatibility guarantees. The correct fix in this situation just might be to fork those clients and correctly add backward compatibility with legacy ES versions.

For an immediate path forward - will put up a small change to support an optional config param in opensearch.yml.  The presence of this parameter controls which version is returned to external clients here. If the parameter is not present (default behavior) we return the current OpenSearch version.

For an immediate path forward - will put up a small change to support an optional config param in opensearch.yml. The presence of this parameter controls which version is returned to external clients here. If the parameter is not present (default behavior) we return the current OpenSearch version.

Will this be a node-local or a cluster-wide setting?

For an immediate path forward - will put up a small change to support an optional config param in opensearch.yml. The presence of this parameter controls which version is returned to external clients here. If the parameter is not present (default behavior) we return the current OpenSearch version.

This sounds like a good approach. In addition, I suggest that we either disable the distribution property, or assign it a different value than opensearch.
This way, clients can start to support different variations using code along these lines.
if version.distribution == 'opensearch':
  if version.opensearchVersion.startsWith('1'):
    compatibleWithElasticsearch7();
  if version.opensearchVersion.startsWith('2'):
    compatibleWithOpensearch2();
else:
  if version.number.startsWith('7'):
    compatibleWithElasticsearch7();
  if version.number.startsWith('6'):
    compatibleWithElasticsearch6();

Will this be a node-local or a cluster-wide setting?

After poking at this a bit, I think it makes sense to be a cluster wide setting to protect against inconsistencies across nodes & the convenience of setting it once with the update settings API during an upgrade.  I threw up an example of this here
@dlvenable I'm not sure I'm following your suggestion for the distribution property.  This was introduced with opensearch here.  If not an opensearch distribution, it should be empty / another value allowing the client code you are suggesting?

Reading through this, I'm wondering if the YML update is the only way we want to support setting versions. Having a REST API would be convenient as it would not require cluster restarts. There would probably need to be some way to durably persist the value set in the event the cluster would restart. Alternatively we could just read off the YML as a baseline after restarts. Just a thought.

@mch2 ,
What I was suggesting was that when we are in backward compatibility mode, we return null or some other constant for distribution.
e.g.
version: {
  "number" : "7.10.2",
  "distribution" : null
 ...
}

When not in that mode, we return it exactly as it is presently.
version: {
  "number" : "1.0.0",
  "distribution" : "opensearch"
 ...
}

The reason for this suggestion is to support a migration path toward the new 1.0.0 OpenSearch version.
Clients can write code that alters the meaning of 1 and 7 depending upon the presence of distribution==opensearch.

The distribution change I suggested might not be necessary however since there is presently no OpenSearch 7.
Clients could write code along the lines of:
if distribution == "opensearch":
  if majorNumber == 1 || majorNumber == 7:
    doSomethingForElasticSearch7()
if distribution is null:
  readElasticSearchVersions()

This code would be valid until OpenSearch 7 is released. At that point in time it would be incorrect.

Will this be a node-local or a cluster-wide setting?

After poking at this a bit, I think it makes sense to be a cluster wide setting to protect against inconsistencies across nodes & the convenience of setting it once with the update settings API during an upgrade. I threw up an example of this here

I am wondering is this going to be an advanced control being offered to users to run in BWC mode for the main response (localhost:9200). There are few other APIs which return node version like _cat/nodes, with this change localhost:9200 with return version as 7.10.2 but _cat/nodes will still return version as 1.0.0. @elfisher is this expected or the plan is to change all the API response?
IMO, if it is an advanced control it can even be provided as a system property (that ways it is not a recommended option but still a way out in case user gets stuck like in this case)
Also, what is the plan to make all these clients understand OpenSearch distribution?

@elfisher is this expected or the plan is to change all the API response?

My understanding from this issue is that the goal is to enable users to define their own version numbers for backwards compatibility/client compatibility where version checking could happen. If the version number is returned in multiple routes my instinct is to make that value consistent across all routes. Inconsistencies with version numbers across different API calls might cause unintended bugs/side effects. Is the version value used in the different API calls fetched some sort of global variable?

Some random thoughts.
I would be very scared of a global, opensearch.version setting because it implies too many things. However, if we are confident that we can implement a complete 7.10.2 backwards compatibility mode that quacks like 7.10.2, then I think a global opensearch.compatibility.version = 7.10.2 setting could make sense, it would not allow random values (only 7.10.2 and any future version we want to be compatible with) and _cat/nodes would then need to also return 7.10.2 and OpenSearch would need to behave like a 7.10.2 in every possible way. It might do it today and I think provides a good path to be forward compatible with 8.x if we chose to.
I have mixed feelings to be allowing any random version number in opensearch.yml, but I can see how it can be convenient if we only want to change the number in the REST API. If that's the case, I'd go with the most specific key to what's being changed, e.g. opensearch.rest.api.version.number = 7.10.2. That doesn't imply OpenSearch is running as 7.10.2, it just says that the API returns 7.10.2. It's a lot less work, but I suspect we might regret this choice and want to do the above at some point.

However, if we are confident that we can implement a complete 7.10.2 backwards compatibility mode that quacks like 7.10.2, then I think a global opensearch.compatibility.version = 7.10.2

I do think that this could be a very valuable field to include. It would provide the solution for clients going forward. Shall I open a ticket to discuss this idea further?
That idea would help with how clients are written going forward. For current versions of clients, we would still need to return version.number: 7.10.2 in order for them to work without modification.

However, if we are confident that we can implement a complete 7.10.2 backwards compatibility mode that quacks like 7.10.2, then I think a global opensearch.compatibility.version = 7.10.2 setting could make sense, it would not allow random values (only 7.10.2 and any future version we want to be compatible with) and _cat/nodes would then need to also return 7.10.2 and OpenSearch would need to behave like a 7.10.2 in every possible way.

It makes sense to me that 7.10.2 would be what is targeted from a backwards compatibility perspective. My fear of not having consistent version numbers returned across the API is that clients/tools may be validating against version numbers in more than one place. The claim in FAQ 1.8 on the website is that it API responses should "quack" like 7.10. Is there more than just version number that would cause it to not?

I do think that this could be a very valuable field to include. It would provide the solution for clients going forward. Shall I open a ticket to discuss this idea further?

No need, this is the issue and we're basically talking what the best implementation would be. Is someone already writing the code? @shwetathareja @nknize?

Let's not overthink this hack. It's a stopgap solution for 1.0+ forward compatibility with existing legacy clients to prevent users from having to mass upgrade.

The distribution change I suggested might not be necessary

It's not necessary but what your suggesting isn't a bad idea to spoof MainResponse exactly like a legacy cluster. This could be done in a follow up PR / "enhancement".

my instinct is to make that value consistent across all routes

/cat API uses the transport layer. Changing it breaks nodes joining the cluster so we don't want to do this.

I have mixed feelings to be allowing any random version number in opensearch.yml, but I can see how it can be convenient if we only want to change the number in the REST API.

Yes. This approach is a forward compatibility hack for MainResponse content until a compatibility module can do this "correctly". I would suggest that compatibility module provide an endpoint to return the spoofed version information instead of :9200 returning a lie.

Also, what is the plan to make all these clients understand OpenSearch distribution?

If OpenSearch sticks with Elastic clients then compatibility going forward is an unknown. Case in point: the new java client which Elastic intends to replace the existing Java High Level rest client. At some point here (preferably before 2.0) OpenSearch will be forced to fork the clients to ensure compatibility with a diverging core.

Having a REST API would be convenient as it would not require cluster restarts.

Rolling upgrades are required to deploy OpenSearch anyway.

Assigned to @mch2

Put up a draft here for some initial comments

Hi, do we have any idea when we will have this released?
Just tried latest opensearch docker image (with tag 1.0.0-rc1), still return below version number:
{
  "name" : "6d33cb208961",
  "cluster_name" : "es-cluster",
  "cluster_uuid" : "Y034pkVIRxq7E4DtbPH6Kg",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.0.0-rc1",
    "build_type" : "tar",
    "build_hash" : "26d579287f50bb33e17c8fe1f05ea208d5c64d1f",
    "build_date" : "2021-05-28T18:18:49.848386Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  }
}

And when logstash started, still report below errors:
logstash            | [2021-06-07T23:08:25,184][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'https://es:9200/_template/logstash'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError" ...

The upcoming change should certainly help allow easy migration to using OpenSearch with existing clients.
I also want to bring up the topic of how compatibility will be expressed going forward. Once an ES administrator flips the switch off, clients will need to examine at the version.distribution and version.number to determine how to communicate with the OpenSearch cluster. This means clients must have built-in mappings from each OpenSearch version to compatible ElasticSearch versions.
For example, clients will need to know OpenSearch 1.0 is compatible with ElasticSearch 7.10, OpenSearch 2.0 is compatible with ElasticSearch 8.0 (hypothetically), OpenSearch 3.0 is compatible with ElasticSearch 8.5 (again, hypothetical), or whatever it is.
I'm hoping to open the discussion on how what support OpenSearch should provide for advertising ElasticSearch compatibility with clients. OpenSearch could make it easier for clients to know which ElasticSearch version the current OpenSearch version is compatible with. One possible solution is adding a new property named version.compatibility_number which would have the 7.10.2.
For example, curl http://localhost:9200 would have:
"version" : {
    "distribution" : "opensearch",
    "number" : "1.0.0-SNAPSHOT",
    "compatibility_number" : "7.10"

This can make it much easier for clients to support both ElasticSearch and OpenSearch going forward.
I think one big question is whether OpenSearch will retain compatibility with ElasticSearch or not. If not, then something like this makes little sense. But if so, then this could be a good way to encourage clients to write code for both ElasticSearch and OpenSearch.

I think one big question is whether OpenSearch will retain compatibility with ElasticSearch or not. If not, then something like this makes little sense. But if so, then this could be a good way to encourage clients to write code for both ElasticSearch and OpenSearch.

OpenSearch is derived from Elasticsearch. It's a fork at 7.10.2, so it's only compatible with ES until then. I expect ES to diverge. Clients can decide whether they want to support either or both.
We tried to answer some of these questions in the upgrading FAQ. If we didn't do a good job, maybe bring more questions up? Try to answer them in a PR and we can discuss!

We tried to answer some of these questions in the upgrading FAQ. If we didn't do a good job, maybe bring more questions up? Try to answer them in a PR and we can discuss!

@dblock FAQ link is broken

@shwetathareja Works for me... where does it land you? https://opensearch.org/faq/#c3

@dblock , @nknize
I want to bring up the idea of the distribution property again so that clients can know how to handle version going forward.
With the merge of #847 , if the setting is activated, then clients should get this:
"version" : {
    "distribution" : "opensearch",
    "number" : "7.10.2",

I believe that we want our clients to write code that can support all of these possibilities: ElasticSearch 7, OpenSearch 1 with version compatibility, and OpenSearch 1 without version compatibility. This helps with the migration path to OpenSearch 1 without version compatibility.
Currently, when in version compatibility mode, the client will have to add some short-term hack that also assumes OpenSearch 7 is the same as OpenSearch 1. Clients can write cleaner, more-robust code if we just remove the distribution value, or give it some other constant name, like version-compatible-opensearch. Then, OpenSearch 1 with version compatibility is treated like ElasticSearch 7 by the client.

I see your point @dlvenable, we're going to have a problem when we release OpenSearch 7.10.2. WDYT @nknize @mch2?

I think by the time OpenSearch 7.10.2 is released there will be a long tail divergence from the Elasticsearch codebase that this "compatibility" issue will be a distant memory. But this is a good point.
The distribution key-value was added as a way of tracking compatibility and "forked" distributions. It's a new field in OpenSearch that legacy clients (forward compatibilty) will not even be aware of so there should be no issue.  Nevertheless, for completeness we could simply not set this field when running in "compatibility" mode so that MainResponse looks exactly like the legacy response.

I opened https://github.com/opensearch-project/OpenSearch/issues/880

Yes, because ElasticSearch does not have the distribution property, that is why I think we can remove it in version compatibility mode. Ticket #880 is what I suggested and should help clients write code supporting different scenarios. Thanks!

Are you going to release a new RC in July with the fix? My team can't use filebeat in production

A cluster setting was introduced in https://github.com/opensearch-project/OpenSearch/pull/847. We removed the distribution in backwards-compatible mode in https://github.com/opensearch-project/OpenSearch/pull/898. Set compatibility.override_main_response_version to put the cluster into a configuration that returns 7.10.2 as with ES OSS.
I am going to close this because the code is done. I added an issue to document it in https://github.com/opensearch-project/documentation-website/issues/72.

Are you going to release a new RC in July with the fix? My team can't use filebeat in production

This will be in 1.0 GA, shortly.


sheet_to_json always returning formatted values

How do I access raw values? We need floating point precison and sheet_to_json return truncated floats.
Looks like I can access raw value by workbook.Sheets[name]['I3'].v but no format option to pass in sheet_to_json method.

The sheet_to_json function takes an options argument.  If you pass {raw:true} as the second argument it will give you the raw values.


worry~  why Only one CPU is used

Why is my mongodb server only one CPU is used?

I'm not a mongodb server developer.

Thank you for your quick reply,
How do I use master-slave?
mongodb:masterSlave(mongodb_master, "<IP_ADDRESS>:27017", "<IP_ADDRESS>:27017"),
mongodb:connect(mongodb_master),


Better handling of repositories that are out of reach

Currently OMRS queries that go through the enterprise connector (ie synchronous) may go out to multiple repositories.
Either 1 or all (1 or more) could be down and hit a timeout.
If we continue to accept inbound requests and contact the failing repository we could start creating a backlog in the system, and/or cause a significant slowdown.
One approach might be to mark a repository as down when a request times out, and then not to contact the repository for a short while - perhaps 10s, 1 minute, 5 minutes (maybe with increasing values each time).
This also brings up the point that in a federated environment we may not always have all the results. We could consider adding something to the response to indicate we know there is an incomplete response.
opening for discussion ....
Migrated from EGERIA-17

This may be related to #3058 and #3300 - in the sense that a single architected approach could potentially satisfy this group of issues.


inet plugin on decode converts float-as-string to inet

I don't have a suggested fix right now, but I'm guessing that ensuring there are at least 4 parts separated by "." for ipv4 and that parts are separated by ":" would be a good check
euneus:decode(<<"{\"test\":\"0.000\",\"ip\":\"<IP_ADDRESS>\"}">>,#{plugins => [inet]}).
{ok,#{<<"ip">> => {10,12,13,14},<<"test">> => {0,0,0,0}}}

Thanks!


Wifi Light compatibility

Hello,
I'm trying to use this plugin to control my Awox striimlight (Wifi).
I don't know if this plugin is compatible but here debug informations when I connect :
2021-01-15 15:09:24 DEBUG (MainThread) [custom_components.awox.config_flow] Processing device - {'objectId': 'UgkMej2G1E', 'address': '', 'modelName': 'SLCW13', 'friendlyName': 'Awox Dimmable Light (SLCW13-2301C9)', 'uuid': '24ab0aae-ea38-4f68-9436-84c1e1d19ed3', 'typedRoom': {'__type': 'Pointer', 'className': 'Room', 'objectId': 'JgpWWKqIuB'}, 'displayName': 'Awox Dimmable Light (SLCW13-2301C9)', 'room': 'Salon', 'owner': {'__type': 'Pointer', 'className': '_User', 'objectId': 'QwRLs3Oxim'}, 'syncable': True, 'provider': 'UPnPLight', 'vendor': 'Awox', 'type': '.wifi.light.switch.color.white.speaker.dimming.temperature.', 'createdAt': '2019-11-15T21:45:00.316Z', 'updatedAt': '2019-11-15T21:45:00.316Z', 'ACL': {'QwRLs3Oxim': {'read': True, 'write': True}}} 2021-01-15 15:09:24 WARNING (MainThread) [custom_components.awox.config_flow] Skipped device, missing macAddress - {'objectId': 'UgkMej2G1E', 'address': '', 'modelName': 'SLCW13', 'friendlyName': 'Awox Dimmable Light (SLCW13-2301C9)', 'uuid': '24ab0aae-ea38-4f68-9436-84c1e1d19ed3', 'typedRoom': {'__type': 'Pointer', 'className': 'Room', 'objectId': 'JgpWWKqIuB'}, 'displayName': 'Awox Dimmable Light (SLCW13-2301C9)', 'room': 'Salon', 'owner': {'__type': 'Pointer', 'className': '_User', 'objectId': 'QwRLs3Oxim'}, 'syncable': True, 'provider': 'UPnPLight', 'vendor': 'Awox', 'type': '.wifi.light.switch.color.white.speaker.dimming.temperature.', 'createdAt': '2019-11-15T21:45:00.316Z', 'updatedAt': '2019-11-15T21:45:00.316Z', 'ACL': {'QwRLs3Oxim': {'read': True, 'write': True}}}
And I get this message from intergration : "[%key:common::config_flow::abort::no_devices_found%]"
Thanks

No the integration only supports some of the bluetooth mesh lights for now.
Only the lights where type starts with .ble.tlmesh currently work. The .ble.zigbee.* protocol also differs see #2
Are the WiFi lights also able to be part of the bluetooth mesh?
Would be happy to look into this, but as I don't have such a wifi light it will be hard for me to get this to work.

No the integration only supports some of the bluetooth mesh lights for now.
Only the lights where type starts with .ble.tlmesh currently work. The .ble.zigbee.* protocol also differs see #2
Are the WiFi lights also able to be part of the bluetooth mesh?
Would be happy to look into this, but as I don't have such a wifi light it will be hard for me to get this to work.

How can I know that Wifi Light is part of the bluetooth mesh ?
I can handle my Wifi Light with Awox Smart Control Android App.
I will try to have a look in the code and see what informations Awox cloud returns.

How can I know that Wifi Light is part of the bluetooth mesh ?
I can handle my Wifi Light with Awox Smart Control Android App.
I will try to have a look in the code and see what informations Awox cloud returns.

Had a look into the code of the Smart Control Android App and it seems this wifi light can't be controlled over Bluetooth.
For now I don't have adding support for these devices on my list, sorry. Mostly because I don't have one my self.
Will add a note to the readme that these wifi lights aren't supported with this integration

Is it not possible to control the lights etc via WIFI using the plug plus switch from Awox? Synology DSM 7 does not support Bluetooth. It should be possible in HA over Wifi because it is now already working via the Awox app and/or Google


problem with todo api

I have followed the instructions in  Dependencies section. I have node v0.12.3 and npm v2.13.2
I see this in my chrome console GET http://localhost:8080/api/todos 404 (Not Found). As the app suggests I have installed these module as well, but still no luck
npm install express connect-history-api-fallback morgan body-parser
Any clues?

yeah I should have added another line. Rather than using npm run server run npm run express for the api to work.

If I do npm run express then app completely fails with below error in chrome console:
GET http://localhost:8080/__build__/common.js
(index):39 GET http://localhost:8080/__build__/angular2.js 404 (Not Found)
(index):41 GET http://localhost:8080/__build__/app-simple.js 404 (Not Found)

can you try  NODE_ENV=development npm run express

that fixes the problem! Thanks for the quick response and keep up the great work!

Can you update the building/instructions so we don't all run into the same issue after installing?

@gsans @gdi2290 This PR https://github.com/angular-class/angular2-webpack-starter/pull/34 also updates the build/instructions to include that fix :+1:


ZEVA - Dashboard Loading Performance Issues for Director Role

Title: ZEVA - Dashboard Loading Performance Issues for Director#1601a
Description:
The Director role is experiencing performance issues when logging in and loading the dashboard. The first screen shows nothing to approve and takes a long time to load with the current status of activity.
Expected Behaviour:
When the user logs-in and when the dashboard loads, the correct status of actions and information should be displayed. While the dashboard is loading, it should be clear to the user that the dashboard is still loading.
Actual Behaviour:
The dashboard loads initially showing no activity information as if there was nothing for the user to do.
Implications:
The user is confused about what actions they have to do in the system.
Steps To Reproduce:
Log in as a Director.
Observe the dashboard loading time and information displayed.

Hey team! Please add your planning poker estimate with Zenhub @AlexZorkin @jig-patel @tim738745 @vibhiquartech @emi-hi @JulianForeman


wav: Fix division by zero for empty stream

Quick fix for an edge case. Might fix #174, didn't test yet.

Finally fixes #174, confirmed by regression test.

Thanks!


dont Work

pls reply wheres the exe file

Just to inform you, this repo is dead, but I'm working on a remake here https://github.com/0PandaDEV/awesome-windows


is this still up?

Maybe would be good to remove it since there's an aspector? It can be confusing when you see two similar repos

Ping ;)

Yeah. I was doing some experiment in this repo a couple months back.

On Nov 18, 2015, at 12:19 PM, Maciej Mensfeld<EMAIL_ADDRESS>wrote:
Ping ;)
—
Reply to this email directly or view it on GitHub https://github.com/gcao/aspect4r/issues/39#issuecomment-157785092.

Agree with @mensfeld here. Got initially confused by this as well.
One thing I'm for instance still not sure about is if Aspector is a 'spiritual' successor of Aspect4r or in what relation these projects stand (fork? independent?). A hint in the README file at least would be helpful.
Thank you.