issue with express 4.17.1 and websockets

I have an issue with express v.4.17.1 and socket.io (but the same thing happens with the ws-package).
Code example:
"use strict";

const express = require('express');
const app = express();
const server = require('http').createServer(app);
const io = require('socket.io')(server);

io.on('connection', (socket) => {
  console.log('user connected', socket.id);
});

server.listen(5000, () => console.log('Server listening on port 5000'));

This should expose a simple websocket server on port 5000. The problem is that the connections get refused. The client reconnects every 2-3 seconds.
By downgrading to express v.4.13.0, everything works fine, it's the only thing I've changed on the code so it is an issue with express I guess?
Anyone encountered this problem?

Hi @samleurs247 that is quite strange, as websockets do not actually need Express to function. For example, you can remove Express from your example app and it works just fine, as Express is not involved. I tried your example with 4.17.1 and did not have any issue. I of course, though, had to make up my own client to do this, and perhaps that is why I was unable to reproduce the issue. Can you provide the complete steps to replicate the issue?


New best practice: reactivity

Add a new best practice around reactivity: if you derive data from a property that the user can update, you should react to changes in that property.
For reference: https://github.com/AlaskaAirlines/auro-banner/issues/27

Related https://github.com/AlaskaAirlines/WC-Generator/issues/224


ɖ  errors on numbers

Vyxal, 5 bytes
123ɖ+

Try it Online!

It should rangify or digitify, y'all decide which


style: update pages design

This PR:

Updates the ItemCard style in Highlights components;

Preview

Preview Home page
Preview Case page

Steps to test

[x] Open the preview pages;
[x] Confirm that everything looks and works as expected;

This is great, can you also add a "Use Cases" drop down at the top with "Security, Networking, Observability" being the choices with links to the relevant sections on the homepage?

This is great, can you also add a "Use Cases" drop down at the top with "Security, Networking, Observability" being the choices with links to the relevant sections on the homepage?

done

Thanks! Last things, can you add an arrow or something else to the cards on the homepage so that people know they should click them? I think we had this before but it got removed in the redesign

Thanks!


Allow for minimum selector depth

Pull Request Template
Description
Want the ability to pass minimum values for data-attribute selectors using the finder methods properties of optimizedMinLength and seedMinLength. We added a new setting called treat data attributes as unique (this will not generate nested selectors)
Type of change
Please delete options that are not relevant.

[x] New feature (non-breaking change which adds functionality)
[x] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes.
Checklist:

[x] My code follows the style guidelines of this project. npm run lint passes with no errors.
[x] I have made corresponding changes to the documentation
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes. npm run test passes with no errors.

Still working on this. Looking at tests

@drodenberg thanks for contributing. From your description I still do not understand what we are actually doing here. Can you provide a clear, point by point description of the use case

the problem you encountered
the solution you suggest

I know only read "i want to tweak some setting" but I have no idea of the context and if this might be useful for others.

Thanks for responding. So the issue we have is that the finder lib is generating selectors with the child combinator > that include styled component classes. These are unreliable because a change of CSS source will re-generate the class name with a new unique id. This new setting makes it possible to generate "shallow" or non-nested selectors based on data attributes which will not be effected by re-generated classes.
const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch()
  const page = await browser.newPage()

  await page.goto('https://demo.ol-kit.com/?view=39.000000,-96.000000,5.00,0.00')

  await page.setViewport({ width: 1680, height: 916 })

  await page.waitForSelector('.MuiPaper-root > .MuiTabs-root > .MuiTabs-scroller > .sc-paXsP > .MuiTabs-flexContainer > [data-testid="LayerPanel.openTab"]')
  await page.click('.MuiPaper-root > .MuiTabs-root > .MuiTabs-scroller > .MuiTabs-flexContainer > [data-testid="LayerPanel.openTab"]')

  await page.waitForSelector('.MuiPaper-root > .MuiTabs-root > .MuiTabs-scroller > .sc-paXsP > .MuiTabs-flexContainer > [data-testid="LayerPanel.openTab"]')
  await page.click('.MuiPaper-root > .MuiTabs-root > .MuiTabs-scroller > .MuiTabs-flexContainer > [data-testid="LayerPanel.openTab"]')

  await browser.close()
})()

Notice the styled component generated classnames contained within those selectors: .sc-paXsP
What we really want is:
const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch()
  const page = await browser.newPage()

  await page.goto('https://demo.ol-kit.com/?view=39.000000,-96.000000,5.00,0.00')

  await page.setViewport({ width: 1680, height: 916 })

  await page.waitForSelector('[data-testid="LayerPanel.openTab"]')
  await page.click('[data-testid="LayerPanel.openTab"]')

  await page.waitForSelector('[data-testid="LayerPanel.openTab"]')
  await page.click('[data-testid="LayerPanel.openTab"]')

  await browser.close()
})()

Let us know if this doesn't make sense

@glenselle This clears things up a lot! Can you turn the PR into a draft until it is ready for review and I will gladly review it.

Hi @drodenberg Do you by chance have some time to look into this.
@tnolet Without this, the CUSTOM DATA ATTRIBUTE feature is doesn't seem very useful.

pinging @ianaya89 to have a look at this

@xmile1 @tnolet we already merged a PR (#130) related to this.
Basically, when data- attribute is in the element (and you set the data- attr option) we will assume the attribute as a unique identifier (the same applies for id).
You will not see nested selectors for those cases.
We are already working on other improvements and we will ship them soon 🤞

Closing due inactivity. Feel free to open it again whenever is ready


AsDataset does not respect number format

I have a spreadsheet with a cell value of<PHONE_NUMBER>26770000.  It is formatted as a number.  If it is formatted as General, it will display scientific notation and lose precision "1.30407E+17".  When exporting the dataset to a file it gets even worse with a value of 1.30E+17.  The AsDataset does not seem to respect the cell formatting causing the dataset to contain bad data.

Hi,
Indeed, AsDataSet ignores the number format, but that doesn't mean it contains bad data. A double is a double. ExcelDataReader returns bitwise identical doubles as Excel uses internally.
If you need to format the output in your program exactly like Excel, you'd have to use the reader APIs (Read(), GetValue(), GetNumberFormat()) and the ExcelNumberFormat library. It works similar to .NET's ToString(), but using Excel's number format syntax instead.


[feature] Visualising Datasource, Table, Slice and Dashboard Use.

New Feature Suggestion for feature. (I'm happy to implement this but wanted people's thoughts first)
In the "Manage" section, a way of visualising the usage of datasources, tables, slices and dashboards. Goals being to identify high and low use tables and slices to prune old or stale ones.
My suggestion would probably be a D3 network diagram (http://bl.ocks.org/mbostock/4062045) where the size of each node is dictated by how much the item gets used.
One extension would also be to show which metrics and columns are getting used the most too.

I love this idea and just wanted to add some thoughts.
Is the activity log of usage (e.g. the database, table, slice, user, datetime of query, etc.) recorded anywhere today?
If that log exists (of if we start recording it), you could open that up for exploration so that admins could ask abstract questions of the data (Who's the top user? During what time of the day do we see the most query activity against table X? etc.).
I like the notion of a specific viz, and I think it'd be cool to display the most important usage stats in something fixed like the network diagram, but it seems easier to expose the event-level stats in an intuitive way and let admins create their own usage dashboard specific to their environment.

We have a nice Superset Usage dashboard at Airbnb where we track usage over time. Weekly active users, daily active user, recent hourly usage, popular viz types, top dashboards over the past 7 days, top slices, number of content creators, ... We should take the time to ship this along with the examples so that people can track adoption ramping up in their environments.

@mistercrunch that sounds good, I think that should ship as default when you run superset init. I think this would be additive to that though. This is more about seeing the connections. For example it's quite hard to find out which dashboards use a particular metric at the moment, so I don't always know if it's safe to remove. That's eventually the use case I want to cover. Thoughts?
@dylburger we do store the logs, but the parts on metrics and columns are currently stored as json rather as individual values. You can see the logs in the access logs menu item.

@alanmcruickshank very cool. My only feedback here would still be: is there a way to expose both the log data (potentially flattening some of the properties stored in the JSON, projecting them as easily-queryable values) and the slice / dashboard metadata in a more accessible way for querying, so that you enable admins to ask any general questions of that data.
Again, I like the idea of your network viz for your use case, but I'm imaging admins asking many arbitrary questions about their instance, and your viz might not target all of them.
Maybe this is a separate story from yours - just wanted to put the thoughts on paper in case they help.

Notice: this issue has been closed because it has been inactive for 466 days. Feel free to comment and request for this issue to be reopened.

Just wondering if there was an implementation of this feature? Thanks in advance!


Remove unused tests

From what I understand the concept of a "virtual" list-view-item
is going away what with Glimmer (tm) making it unnecessary.

@stefanpenner what are your thoughts on this?

We have some plans on reworking these in new context. I think @raytiley will take the reigns on it.
@jonnii I think it would be useful to cut up the acceptance tests and see if any are already passing.

@mmun i suppose you could rewrite these but they would be a complete re-write as i think most of this stuff is gone. the value here (i think) is the tests which scroll around, but i think those are also covered by the scroll-top-test.js.

Which tests are you talking about? I was referring to the acceptance tests in the tests/acceptance/list-view-test.js file.

@mmun just these ones, the recycling-tests. from what i understand all the hackery put into ember-list-view to manage recycling views just comes for free now with glimmer.

i'm sure most of the other tests can and will be re-written, but these ones deal with ListItemView and ReusableListItemView which are long gone.

Ok let me rephrase: I think @krisselden and @raytiley have an idea for how to recycle the recycling-tests.
I was suggesting that it would be helpful if you dug through the list-view-test.js next.

ah, ok. i can do that! will leave this here for now.

:+1:

the multiheight test could usefully be converted into a test of the mixed grid layout which is currently untested.


Git Submodule doesn't update if modified from integrated terminal

Issue Type
Bug
Description
There is a similar issue @ No auto fetch for submodules . This is however, slightly different.
While working in the same workspace I typically commit from the integrated terminal. When I update the main repository changes are shown immediately. However if I make any changes to a submodule, changes are not updated unless VS Code is reloaded.
Is this another issue that should be fixed by configuring an upstream link or is this a bug?
I didn't include System Info as I thought is was superfluous. It can be added if requested.

I didn't include System Info as I thought is was superfluous.

More details are rarely superfluous. Which OS? Which version? Does it reproduce in the latest insider release?
Here's what I see:


feat: improved validation errors handling

To be merged after #422
This PR improves handling of ValidationError when loading local plugin manifests.
Before this PR:

If one local plugin manifest failed validation, none of the local manifests would be loaded. It was impossible to access any local plugin if there was at least one that failed validation.

After this PR:

If at least one plugin manifest fails validation, a WARNING that includes their names is raised.
Running polus.plugins.print_invalid() prints a report of ValidationErrors found in each invalid manifest.

Looking good Camilo. I have couple questions and one last request :

Question1 : how do we handle versions of manifests? It is likely that several versions on the spec exists side-by-side on some point in time. When are the pydantic models created? One way I can think of is to lazy load the models and stored them in some indexed structure when the parsing of a manifest is requested.
Question2 : I did not see the logic for managing several failing manifests but I may just have missed it (i believe before you were populating a structure and you had this method to extract info later, how does it work now?)
Can we have a simple test that demonstrate loading and validating a correct manifest and some bad manifests? I understand we may think it is overkill at this moment but that's the habits we should get into. Each new feature should be preferably backed by some tests to provide replicable and executable documentation.


[Linter] Update files with respect to sort-imports rule

This solves #9252. This PR is an update from a past PR, https://github.com/Azure/azure-sdk-for-js/pull/18567.
What's new?

I've added the sort-imports rule into ./common/tools/eslint-plugin-azure-sdk/.eslintrc.json
I've updated files appropriately after calling rush rebuild and rush lint.

I used the default settings for sort-imports, so if there should be changes to the rule settings, please let me know!

Hi @bzhang0,
Thanks a lot for these changes!
While it is good to do this change and sort imports inside the plugin source code, it does not solve https://github.com/Azure/azure-sdk-for-js/issues/9252 yet. To do so, the PR will need to add this new behavior as part of the plugin rules list. Once this is done, all imports in all packages across the repository can be sorted to satisfy the rule. Please use rush lint to run the linter for all packages.

Oh, I see!  I assumed putting the rule in the eslint's .eslintrc.json file was enough.
I will have these changes fixed. Thanks for letting me know!

I'm getting an error when trying to call rush rebuild after calling rush update.
The error occurs on @azure/keyvault-certificates:
--[ FAILURE: @azure/keyvault-certificates ]---------[ 1 minute 36.3 seconds ]--

Invoking: npm run clean && tsc -p . && npm run build:nodebrowser && api-extractor run --local

><EMAIL_ADDRESS>clean
> rimraf dist-esm dist-test types *.tgz *.log samples/typescript/dist

error TS6053: File '/Users/ben/Documents/repos/azure-sdk-for-js/sdk/keyvault/keyvault-common/types/challengeBasedAuthenticationPolicy.d.ts' not found.
  The file is in the program because:
    Matched by include pattern '../keyvault-common/**/*.ts' in 'tsconfig.json'
error TS6053: File '/Users/ben/Documents/repos/azure-sdk-for-js/sdk/keyvault/keyvault-common/types/index.d.ts' not found.
  ...1 line omitted...
    Matched by include pattern '../keyvault-common/**/*.ts' in 'tsconfig.json'
error TS6053: File '/Users/ben/Documents/repos/azure-sdk-for-js/sdk/keyvault/keyvault-common/types/parseKeyvaultIdentifier.d.ts' not found.
  The file is in the program because:
    Matched by include pattern '../keyvault-common/**/*.ts' in 'tsconfig.json'
error TS6053: File '/Users/ben/Documents/repos/azure-sdk-for-js/sdk/keyvault/keyvault-common/types/parseWWWAuthenticate.d.ts' not found.
  The file is in the program because:
    Matched by include pattern '../keyvault-common/**/*.ts' in 'tsconfig.json'
error TS6053: File '/Users/ben/Documents/repos/azure-sdk-for-js/sdk/keyvault/keyvault-common/types/tracingHelpers.d.ts' not found.
  The file is in the program because:
    Matched by include pattern '../keyvault-common/**/*.ts' in 'tsconfig.json'

These files all exist within my subdirectory. Any idea what the issue could be?

Hey @bzhang0, consider running rush clean to clean any old artifacts first and see if that fixes the issue.

Okay, thanks! What is the process after running rush clean? Would I rush update then rush rebuild?

@bzhang0 yes! if you ran rush update already and no package.json file was updated, you could skip it and just run rush rebuild.

This worked for me, thank you so much!

Sorry to ask one more question, but do you know how to run rush lint for a specific @azure/*? I think the main issues are at @azure/cosmos, but I'm having to run rush lint for the whole project in order to see those specific errors.
Additionally, I'm seeing a section that says x errors and y warnings potentially fixable with the `--fix` option. How do I run the -fix option in rush lint?

do you know how to run rush lint for a specific @azure/*? I think the main issues are at @azure/cosmos, but I'm having to run rush lint for the whole project in order to see those specific errors.

Go to the package directory, e.g. for cosmos that would be sdk/cosmosdb/cosmos, and run rushx lint` there.

Additionally, I'm seeing a section that says x errors and y warnings potentially fixable with the --fix option. How do I run the --fix option in rush lint?

rush lint:fix or rushx lint:fix depending on if you want to run it for all packages or in a particular one inside its directory, similar to how you would run rush lint and rushx lint as I explained above.

Thank you! This really helps 😄

Is it possible to configure our editors to sort these imports when we save a file according to the rule, same as for syntax formatting? There is a built in source action in VS Code for organizing imports, but I'm not sure if it uses the same logic. The best situation will be if our editors can fix the file for us, but (@deyaaeldeen) I'm not sure all of our rules in the tool that support fixers will be amenable to that kind of workflow. Some need to be manually checked to make sure they didn't do anything bad, otherwise we could just run lint:fix on save if it isn't too slow.

@bzhang0 do you have to manually fix some of the files, i.e., rush lint:fix doesn't fix all of the errors  reported by this rule?

@jeremymeng I've been mostly working section to section, but I definitely should run rush lint:fix then work through. But within the sections where I've ran rushx lint:fix, I've been manually fixing the errors not caught by fix. To eliminate the repetition of sorting all of them by hand, I wrote my own script that can sort most of the given imports that are incorrect. I'm happy to share this code (it's written in Java at the moment) and maybe it could be used to further improve the :fix functionality for sort-imports!
On the side, I'm still getting one error after running rush clean, rush update, then rush rebuild. I'm not sure why this is erroring out:
==[ FAILURE: 1 project ]=======================================================

--[ FAILURE: @azure/cosmos ]-----------------------[ 4 minutes 60.0 seconds ]--

dist-esm/src/index.js → dist/index.js...
[!] Error: Could not resolve entry module (dist-esm/src/index.js).
Error: Could not resolve entry module (dist-esm/src/index.js).
    at error (C:\Users\ben\Documents\repos\azure-sdk-for-js\common\temp\node_modules\.pnpm\rollup@1.32.1\node_modules\rollup\dist\shared\node-entry.js:5400:30)
    at C:\Users\ben\Documents\repos\azure-sdk-for-js\common\temp\node_modules\.pnpm\rollup@1.32.1\node_modules\rollup\dist\shared\node-entry.js:12199:20
    at async Promise.all (index 0)
    at async Promise.all (index 0)
    at async Promise.all (index 0)

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>bundle: `rollup -c`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>bundle script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR!      C:\Users\ben\AppData\Roaming\npm-cache\_logs\2021-11-12T00_22_35_487Z-debug.log```

Regardless, I will continue to fix all the formatting! 😄

one quick thing, running rush lint:fix returns the following errors: ERROR: The project [@azure-rest/synapse-access-control] does not define a 'lint:fix' command in the 'scripts' section of its package.json, so I will continue to just run rush lint, then navigate individually into directories. If there's a fix for this, let me know!

@bzhang0 thanks the project should be fixed.
Also I realized that rush lint:fix will use all the rules so not what we want. There seems to be ways to run fix just for one rule though: https://stackoverflow.com/questions/47828664/how-to-fix-just-one-rule-using-eslint

Fixed a lot of files! I ran rush lint and logged the directories/sections that still need to be addressed:

[ ] sdk/storage/storage-blob/
[ ] sdk/core/core-client-lro-rest/
[ ] sdk/keyvault/keyvault-secrets/
[ ] sdk/purview/purview-catalog-rest/src/
[ ] sdk/communication/communication-short-codes/src/
[ ] sdk/formrecognizer/ai-form-recognizer/src/
[ ] sdk/metricsadvisor/ai-metrics-advisor/src/
[ ] sdk/textanalytics/ai-text-analytics/src/
[ ] sdk/communication/communication-chat/src/
[ ] sdk/communication/communication-identity/src/
[ ] sdk/communication/communication-network-traversal/src/
[ ] sdk/communication/communication-phone-numbers/src/
[ ] sdk/communication/communication-sms/src/
[ ] sdk/communication/communication-sms/test
[ ] sdk/tables/data-tables/src/
[ ] sdk/keyvault/keyvault-admin/
[ ] sdk/keyvault/keyvault-certificates/src/
[ ] sdk/keyvault/keyvault-keys/src/
[ ] sdk/remoterendering/mixed-reality-remote-rendering/src/
[ ] sdk/storage/storage-blob-changefeed/src/

Will attempt to finish these soon!
Here is the script that I created to parse and fix all, multiple, and single imports (in most cases). I will update this as well to potentially handle other cases as well. Maybe one day this can be integrated into the --fix functionality!

It seems I'm still unable to properly rush rebuild. Do I need to fetch upstream for this to work?
But as of now, I've fixed around 85% of the errors. Some projects are unable to show linting errors since they're hidden behind other errors, so I will continue to fix them. Additionally, I am unable to run rushx lint in certain directories, such as sdk\agrifood\agrifood-farming-rest. Let me know if there's anything I can do to fix this issue.
Thanks!

It seems I'm still unable to properly rush rebuild. Do I need to fetch upstream for this to work?
But as of now, I've fixed around 85% of the errors. Some projects are unable to show linting errors since they're hidden behind other errors, so I will continue to fix them. Additionally, I am unable to run rushx lint in certain directories, such as sdk\agrifood\agrifood-farming-rest. Let me know if there's anything I can do to fix this issue.
Thanks!

@bzhang0 I can help looking into agrifood-farming-rest. What error are you getting when running rushx lint in there?

What error are you getting when running rushx lint in there?

It seems sdk\agrifood\agrifood-farming-rest works now (at least on my laptop) 😅
A couple more directories to resolve:

[ ] sdk/storage/storage-file-datalake/
[ ] sdk/storage/storage-file-share/
[ ] sdk/agrifood/agrifood-farming-rest/

Is there anything I can do about my rush rebuild errors? It is shown similarly above.

At long last, all rush lint errors have been fixed! However, rush rebuild still is not working for me.
==[ BLOCKED: 1 project ]=======================================================

These projects were blocked by dependencies that failed:
  @azure-tests/perf-storage-file-datalake

==[ FAILURE: 3 projects ]======================================================

--[ FAILURE: @azure/storage-file-datalake ]---------[ 6 minutes 4.4 seconds ]--

Invoking: npm run clean && tsc -p . && npm run build:nodebrowser && api-extractor run --local && npm run build:types

><EMAIL_ADDRESS>clean
> rimraf dist dist-* typings temp statistics.html coverage coverage-browser .nyc_output *.tgz *.log test*.xml TEST*.xml

><EMAIL_ADDRESS>build:nodebrowser
> rollup -c 2>&1

  ...8 lines omitted...
Error: '@azure/storage-blob' is imported by dist-esm/storage-file-datalake/src/clients.js, but could not be resolved – treating it as an external dependency
    at Object.onwarn (/Users/ben/Documents/repos/azure-sdk-for-js/sdk/storage/storage-file-datalake/rollup.config.js:197:15)
    at Object.onwarn (/Users/ben/Documents/repos/azure-sdk-for-js/common/temp/node_modules/.pnpm/rollup@1.32.1/node_modules/rollup/dist/shared/node-entry.js:13599:25)
    at Graph.onwarn (/Users/ben/Documents/repos/azure-sdk-for-js/common/temp/node_modules/.pnpm/rollup@1.32.1/node_modules/rollup/dist/shared/node-entry.js:13599:25)
    at Graph.warn (/Users/ben/Documents/repos/azure-sdk-for-js/common/temp/node_modules/.pnpm/rollup@1.32.1/node_modules/rollup/dist/shared/node-entry.js:13403:14)
    at ModuleLoader.handleResolveId (/Users/ben/Documents/repos/azure-sdk-for-js/common/temp/node_modules/.pnpm/rollup@1.32.1/node_modules/rollup/dist/shared/node-entry.js:12412:24)
    at ModuleLoader.<anonymous> (/Users/ben/Documents/repos/azure-sdk-for-js/common/temp/node_modules/.pnpm/rollup@1.32.1/node_modules/rollup/dist/shared/node-entry.js:12298:30)
    at Generator.next (<anonymous>)
    at fulfilled (/Users/ben/Documents/repos/azure-sdk-for-js/common/temp/node_modules/.pnpm/rollup@1.32.1/node_modules/rollup/dist/shared/node-entry.js:38:28)

--[ FAILURE: @azure/container-registry ]-----------[ 2 minutes 14.7 seconds ]--

Invoking: npm run clean && tsc -p . && rollup -c 2>&1 && api-extractor run --local

><EMAIL_ADDRESS>clean
> rimraf dist dist-* temp types *.tgz *.log

src/containerRegistryTokenCredential.ts(4,1): error TS6133: 'Mappers' is declared but its value is never read.
src/containerRegistryTokenCredential.ts(5,1): error TS6133: 'Parameters' is declared but its value is never read.
src/containerRegistryTokenCredential.ts(7,10): error TS6133: 'AcrAccessToken' is declared but its value is never read.
src/containerRegistryTokenCredential.ts(7,26): error TS6133: 'AcrRefreshToken' is declared but its value is never read.
src/containerRegistryTokenCredential.ts(8,1): error TS6192: All imports in import declaration are unused.

--[ FAILURE: @azure/cosmos ]-----------------------[ 8 minutes 54.3 seconds ]--

dist-esm/src/index.js → dist/index.js...
[!] Error: Could not resolve entry module (dist-esm/src/index.js).
Error: Could not resolve entry module (dist-esm/src/index.js).
    at error (/Users/ben/Documents/repos/azure-sdk-for-js/common/temp/node_modules/.pnpm/rollup@1.32.1/node_modules/rollup/dist/shared/node-entry.js:5400:30)
    at /Users/ben/Documents/repos/azure-sdk-for-js/common/temp/node_modules/.pnpm/rollup@1.32.1/node_modules/rollup/dist/shared/node-entry.js:12199:20
    at async Promise.all (index 0)
    at async Promise.all (index 0)
    at async Promise.all (index 0)

Projects failed to build.

rush rebuild (9 minutes 51.3 seconds)

Let me know if I'm missing something!

@bzhang0 did you try rush update after merging main?

@bzhang0 did you try rush update after merging main?

I will try this!

@bzhang0 Since this rule impacts almost every package this PR is becoming too large to manage reviews.  We recommend breaking it into smaller ones in phases

First PR to add the rule to the plugin but treat violations as warnings
Then separate PRs for fixing the linter rule in each package.
Once all of them are merged, another PR to change violations from warnings to error

Sorry for the trouble @bzhang0 ! Hopefully you just need to git cherry-pick into new branches then open new PRs.

Will do! Just to clarify, you want me to do the following?

make a new branch + pr that adds the rule with warnings
make a new branch + pr for a subdirectory (whose changes can be pulled in from git cherry-pick)
repeat for each subdirectory

Closing this as we will have separate PRs for packages


Setup gh-action to build docs from docs/_site

Quartodoc puts all files for the site build into docs/_site, but gh-actions can only read from docs, so I currently always copy the content of docs/_site manually into docs. A gh-action like this one  would of course be much more convenient =)

This is a great idea. I recently implemented this with statsframe (very-pre-alpha package) https://github.com/NKeleher/statsframe/blob/main/.github/workflows/ci.yml and heavily influenced by the pins-python workflow: https://github.com/rstudio/pins-python/blob/main/.github/workflows/ci.yml.
Do you have a dependency graph in mind, i.e. "Build tests" >> "Build docs" >> "Publish docs" on merge to main?

Do you have a dependency graph in mind, i.e. "Build tests" >> "Build docs" >> "Publish docs" on merge to main?
This dependency graph graph makes excellent sense to me =)

Perfect, I'll submit a PR with some suggestions on the ci workflow.

Amazing, thank you!


Cannot deploy Azure Machine Learning online endpoint using Cli

I was trying to create an endpoint and deployment in Cloud Shell and Azure Pipelines using the commands from this documentation
https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-mlflow-models-online-endpoints?tabs=endpoint%2Cstudio
My commands are :
 az ml online-endpoint create  --resource-group $(databricks.resourcegroup) --workspace-name $(databricks.workspace.name) --file create-endpoint.yml
 az ml online-deployment create --name deployment --endpoint model-endpoint -f endpoint-config.yml --all-traffic
However, I get the errors:
'online-endpoint' is misspelled or not recognized by the system.
'online-deployment' is misspelled or not recognized by the system.
I am using the following versions:
{
  "azure-cli": "2.33.1",
  "azure-cli-core": "2.33.1",
  "azure-cli-telemetry": "1.0.6",
  "extensions": {
    "ai-examples": "0.2.5",
    "azure-cli-ml": "1.37.0",
    "ssh": "1.0.0"
  }

Is this a bug?

Az ml and azi-cli-ml https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-cli is a conflicting extension -- see this doc to troubleshoot

route to CXP team

@migdaepp Please look into the above suggestions.  Additionally, you could try to upgrade the ML extension to the latest version by using az extension update -n ml.


Don't trust the external cache when looking for field descriptors

When looking for field descriptors for a typeref, we should iterate over every image info available, even those that the cache claimed to have processed, as a last ditch effort.
rdar://114567246
(cherry picked from commit a6be181ae5fe1d388624338ff1a591b615052a66)

@swift-ci test


Add examples about background tasks usage with responses

PR Checklist

[x] Have you followed the guidelines in CONTRIBUTING.md?
[x] Have you got 100% test coverage on new code?
[x] Have you updated the prose documentation?
[x] Have you updated the reference documentation?

The reason why I opened this PR is that I did not understand how to pass route handler arguments to background tasks, but discussion #765 was helpful, and I thought that adding a few examples, as suggested within the discussion, might help other users.
This PR contains the following items:

Moved existing example into a separate file.
Added tests for new and previous examples.
Name of kwarg was incorrectly documented as background_task, it should be background, if I am not mistaken.
Given more visibility to new examples, because I thought that the use case in which a background task has to receive route handler arguments might be more common. I also emphasized the fact that background tasks passed into route decorators cannot use handler arguments.

Of course, I am open to any feedback, in particular about the fourth point.
Thank you very much,
Goodbye,
Alessio

@all-contributors add @pomma89 doc


Unable to create a directory

What happened:
Trying to create a directory in a bucket and it silently fails.
What you expected to happen:
Expect a key created in a bucket.
Minimal Complete Verifiable Example:
from s3fs import S3FileSystem
s3 = S3FileSystem(anon=False)
s3.makedirs('s3://my-bucket-name/testdir', exist_ok=False) #fails to create
s3.makedirs('my-bucket-name/testdir', exist_ok=False) #fails to create

How do I create a directory in a bucket? Am I doing something wrong or is it a bug? I see conversation in #245 and I was wondering if you could please help explain what the intended behavior is.
Anything else we need to know?:
s3fs version == 0.5.0
Relevant issue here: #245
Environment:

Dask version:
Python version: 3.7.7
Operating System: Ubuntu Linux
Install method (conda, pip, source): conda

This is a long-standing misunderstanding, and your help, @srib , would be greatly appreciated in clarifying the situation for other users.
s3 does not support directories, except the top-level buckets, which are fairly similar in their behaviour. If you expect s3.makedirs('my-bucket-name/testdir') to enable you to write files with paths that start with that string then: congratulations! There is in fact nothing for s3fs to do, the location was already writable.
Once you have a file like "my-bucket-name/testdir/testfile", the directory will appear to exist, without that file it will not. This is similar to trying to commit an empty directory to git.
Unfortunately, the situation is complicated by the behaviour of the s3 console, which has a "make directory" button, whose action is actually to create an empty file with the name of the directory you wanted. This is not part of the s3 API, just a convention, and the AWS CLI does not do this. If we were to do this, we end up making extra requests, and have to distinguish between real common preixes (directories implied by having contents) and empty placeholder files.

@martindurant AWS have a concept of S3 folders. Check https://docs.aws.amazon.com/AmazonS3/latest/user-guide/using-folders.html
I am working on a comprehensive Pull request to have it as a feature in s3fs.
Check https://github.com/spsoni/s3fs/tree/feature/issue_401_support_for_s3_directory for my WIP

@martindurant AWS have a concept of S3 folders. Check https://docs.aws.amazon.com/AmazonS3/latest/user-guide/using-folders.html
I am working on a comprehensive Pull request to have it as a feature in s3fs.
Check https://github.com/spsoni/s3fs/tree/feature/issue_401_support_for_s3_directory for my WIP

The document you reference makes very clear that this concept of folder is relevant only to the AWS console view of S3, and not inherent to S3 and not implemented/supported by the AWS CLI, AWS SDKs, or REST API.

It is perfectly acceptable to have a key with trailing "/" on S3 which contains data and is not a folded, but it will not be accessible via the console
you do not need intermediate pseudo-direcories to exist when creating a nested key:

s3.exists("bucket/dir") # False, with or without "/"
s3.touch("bucket/dir/inner/afile")  # this is allowed
s3.exists("bucket/dir") # Should be True, even though there is no such key with or without "/"

I think that, for the sake of this repo, we want to maintain adherence to the the true nature of S3 rather than whatever the console does. However, this thing has come up frequently, and s3fs already does make some assumptions around the meaning of "/" - I agree that we should think clearly about how best to handle and describe folder operations.

The document you reference makes very clear that this concept of folder is relevant only to the AWS console view of S3, and not inherent to S3 and not implemented/supported by the AWS CLI, AWS SDKs, or REST API.

It is perfectly acceptable to have a key with trailing "/" on S3 which contains data and is not a folded, but it will not be accessible via the console
you do not need intermediate pseudo-direcories to exist when creating a nested key:

s3.exists("bucket/dir") # False, with or without "/"
s3.touch("bucket/dir/inner/afile")  # this is allowed
s3.exists("bucket/dir") # Should be True, even though there is no such key with or without "/"

I think that, for the sake of this repo, we want to maintain adherence to the the true nature of S3 rather than whatever the console does. However, this thing has come up frequently, and s3fs already does make some assumptions around the meaning of "/" - I agree that we should think clearly about how best to handle and describe folder operations.

@martindurant AWS doc (important section) says following

The Amazon S3 console treats all objects that have a forward slash "/" character as the last (trailing) character in the key name as a folder, for example examplekeyname/. You can't upload an object that has a key name with a trailing "/" character using the Amazon S3 console. However, you can upload objects that are named with a trailing "/" with the Amazon S3 API by using the AWS CLI, AWS SDKs, or REST API.

So it's not only a feature for AWS console but also a S3 core feature accessible via S3 API (CLI, SDK and REST API)
I agree we should not create pseudo-directories (intermediate ones), but if asked/expected to create an explicit folder object (without file objects), s3fs should honour s3.mkdir and offer the mechanism.
E.g.
s3.mkdir("bucket/intermediate-dir/dir1")
Should create "intermediate-dir/dir1/" key in "bucket", but not the "intermediate-dir/"`
Two reasons why we should implement and adhere to S3 API for folders:

So when AWS says that it accepts keys ending with "/" as a folder, then s3fs should comply with the expectation.
s3. mkdir followed by s3.exists should return True from the API design point of view. And S3 API supports a mechanism to create and validate folder. You don't need to have objects inside this folder for its validation of existence.
Explicit folder key ending with "/" may not be an efficient use of S3, but it exists as a feature to create a place holder real folder object instead of common virtual folder objects

s3.mkdir("bucket/dir") # creates key "dir/" in "bucket" s3.exists("bucket/dir") # True, with or without "/" s3.touch("bucket/dir/inner/afile")  # this is allowed s3.exists("bucket/dir") # Should be True, even though there is no such key with or without "/"

@martindurant AWS doc (important section) says following

The Amazon S3 console treats all objects that have a forward slash "/" character as the last (trailing) character in the key name as a folder, for example examplekeyname/. You can't upload an object that has a key name with a trailing "/" character using the Amazon S3 console. However, you can upload objects that are named with a trailing "/" with the Amazon S3 API by using the AWS CLI, AWS SDKs, or REST API.

So it's not only a feature for AWS console but also a S3 core feature accessible via S3 API (CLI, SDK and REST API)
I agree we should not create pseudo-directories (intermediate ones), but if asked/expected to create an explicit folder object (without file objects), s3fs should honour s3.mkdir and offer the mechanism.
E.g.
s3.mkdir("bucket/intermediate-dir/dir1")
Should create "intermediate-dir/dir1/" key in "bucket", but not the "intermediate-dir/"`
Two reasons why we should implement and adhere to S3 API for folders:

So when AWS says that it accepts keys ending with "/" as a folder, then s3fs should comply with the expectation.
s3. mkdir followed by s3.exists should return True from the API design point of view. And S3 API supports a mechanism to create and validate folder. You don't need to have objects inside this folder for its validation of existence.
Explicit folder key ending with "/" may not be an efficient use of S3, but it exists as a feature to create a place holder real folder object instead of common virtual folder objects

s3.mkdir("bucket/dir") # creates key "dir/" in "bucket" s3.exists("bucket/dir") # True, with or without "/" s3.touch("bucket/dir/inner/afile")  # this is allowed s3.exists("bucket/dir") # Should be True, even though there is no such key with or without "/"

The Amazon S3 console treats all objects that have a forward slash "/" character as the last (trailing) character in the key name as a folder, for example examplekeyname/. You can't upload an object that has a key name with a trailing "/" character using the Amazon S3 console. However, you can upload objects that are named with a trailing "/" with the Amazon S3 API by using the AWS CLI, AWS SDKs, or REST API.

So it's not only a feature for AWS console but also a S3 core feature accessible via S3 API (CLI, SDK and REST API)

I'm sorry, but the quoted text says the exact opposite of your interpretation: the special behaviour applies to the console only, and from CLI, SDK, REST, the "/" character is not special.
I also invite you to consider what you think the right behaviour ought to be in cases where:

keys named "/bucket/inner" and "/bucket/inner/" both exist
keys named "/bucket/inner" and "/bucket/inner/stuff" both exist
a key exists at "/bucket/inner/" which contains data and/or metadata

s3. mkdir followed by s3.exists should return True

This point I agree is a problem, and there is an issue for this; but I think it's OK to say in documentation that mkdir is a no-op except for creating buckets. You could solve the issue by keeping tack of mkdirs within the instance, not touching the remote storage backend.

The Amazon S3 console treats all objects that have a forward slash "/" character as the last (trailing) character in the key name as a folder, for example examplekeyname/. You can't upload an object that has a key name with a trailing "/" character using the Amazon S3 console. However, you can upload objects that are named with a trailing "/" with the Amazon S3 API by using the AWS CLI, AWS SDKs, or REST API.

So it's not only a feature for AWS console but also a S3 core feature accessible via S3 API (CLI, SDK and REST API)

I'm sorry, but the quoted text says the exact opposite of your interpretation: the special behaviour applies to the console only, and from CLI, SDK, REST, the "/" character is not special.
I also invite you to consider what you think the right behaviour ought to be in cases where:

keys named "/bucket/inner" and "/bucket/inner/" both exist
keys named "/bucket/inner" and "/bucket/inner/stuff" both exist
a key exists at "/bucket/inner/" which contains data and/or metadata

s3. mkdir followed by s3.exists should return True

This point I agree is a problem, and there is an issue for this; but I think it's OK to say in documentation that mkdir is a no-op except for creating buckets. You could solve the issue by keeping tack of mkdirs within the instance, not touching the remote storage backend.

Very thought-provoking scenarios.
That raises a fundamental question. Is the objective of the s3fs project is to wrap what AWS S3 API offers or we keep the API interface for s3fs compliant with a general-purpose file system like Linux/POSIX fs.
My understanding is that s3fs is just a wrapper around the boto3 library and should offer what AWS S3 API offers and capable of.
I will try to answer with that assumption in mind. Remove "/bucket" from strings where I refer to the boto3 key below.

keys named "/bucket/inner" and "/bucket/inner/" both exist

"/bucket/inner" should be able to represent both file and directory from the s3fs API design point of view.
But if two keys exist "/bucket/inner" (file) and "/bucket/inner/" (directory) then
s3.isdir("/bucket/inner") # and
s3.isfile("/bucket/inner") # both returns True

And, explicit check for directory key
s3.isdir("/bucket/inner/") # returns True
s3.isfile("/bucket/inner/") # returns False

Deletion scenarios
s3.rm("/bucket/inner")  # deletes file key

s3.rmdir("/bucket/inner") # and,
s3.rmdir("/bucket/inner/")  # deletes the directory

Creation scenarios
s3.touch("/bucket/inner")  # creates a file

s3.mkdir("/bucket/inner") # and
s3. kdir("/bucket/inner/")  # creates a directory with key: "/bucket/inner/"

> keys named "/bucket/inner" and "/bucket/inner/stuff" both exist

"/bucket/inner" could be anything again (file and/or directory)
And, "/bucket/inner/stuff" should be okay to co-exist in all of the scenarios for "/bucket/inner" (as a file and/or directory)

s3.touch("/bucker/inner")  # creates a file
s3.mkdir("/bucket/inner")  # creates a directory key as well (key: "/bucker/inner/")
s3.touch("/buckker/inner/stuff")  # creates a file (can try to create the directory ("stuff") as well

Now expected outcome for validation

s3.isfile("/bucket/inner")  # returns True
s3.isdir("/bucket/inner")  # return True
s3.isfile("/bucket/inner/")  # returns False
s3.isdir("/bucket/inner/")  # returns True
s3.isfile("/bucker/inner/stuff")  # returns True

The most challenging part is the deletion

s3.rm("/bucker/inner")  # should delete the file key
s3.rmdir("/bucket/inner")  # should raise an exception that directory is not empty
s3.rmdir("/bucker/inner", recursive=True)  # should delete both keys "/bucker/inner/" and "/bucker/inner/stuff"

> a key exists at "/bucket/inner/" which contains data and/or metadata

boto3 allows storing data for the "/bucket/inner/" key. Check the following output (highlighted directory content size)

% aws s3 ls s3://bucket/
PRE inner/
2021-01-07 19:00:01          0 inner
% aws s3 ls s3://bucket/inner/
2021-01-07 18:58:05          0
2021-01-07 18:57:38          0 stuff
% aws s3 ls s3://bucket/inner/

boto3_s3_client.put_object(Bucket='bucket', Key='inner1/', Body="sdf")
boto3_s3_client.put_object(Bucket='bucket', Key='inner1/stuff', Body="sdf")

% aws s3 ls s3://bucket/
PRE inner1/
% aws s3 ls s3://bucket/inner1/
2021-01-07 19:28:11          3
2021-01-07 19:28:47          3 stuff

> You could solve the issue by keeping track of mkdirs within the instance, not touching the remote storage backend.

I would not make an assumption for end-user of the s3fs library. s3fs is just a wrapper library end of the day. And documenting that mkdir is a no-op breaks that rule of the wrapper library responsibility

Very thought-provoking scenarios.
That raises a fundamental question. Is the objective of the s3fs project is to wrap what AWS S3 API offers or we keep the API interface for s3fs compliant with a general-purpose file system like Linux/POSIX fs.
My understanding is that s3fs is just a wrapper around the boto3 library and should offer what AWS S3 API offers and capable of.
I will try to answer with that assumption in mind. Remove "/bucket" from strings where I refer to the boto3 key below.

keys named "/bucket/inner" and "/bucket/inner/" both exist

"/bucket/inner" should be able to represent both file and directory from the s3fs API design point of view.
But if two keys exist "/bucket/inner" (file) and "/bucket/inner/" (directory) then
s3.isdir("/bucket/inner") # and
s3.isfile("/bucket/inner") # both returns True

And, explicit check for directory key
s3.isdir("/bucket/inner/") # returns True
s3.isfile("/bucket/inner/") # returns False

Deletion scenarios
s3.rm("/bucket/inner")  # deletes file key

s3.rmdir("/bucket/inner") # and,
s3.rmdir("/bucket/inner/")  # deletes the directory

Creation scenarios
s3.touch("/bucket/inner")  # creates a file

s3.mkdir("/bucket/inner") # and
s3. kdir("/bucket/inner/")  # creates a directory with key: "/bucket/inner/"

> keys named "/bucket/inner" and "/bucket/inner/stuff" both exist

"/bucket/inner" could be anything again (file and/or directory)
And, "/bucket/inner/stuff" should be okay to co-exist in all of the scenarios for "/bucket/inner" (as a file and/or directory)

s3.touch("/bucker/inner")  # creates a file
s3.mkdir("/bucket/inner")  # creates a directory key as well (key: "/bucker/inner/")
s3.touch("/buckker/inner/stuff")  # creates a file (can try to create the directory ("stuff") as well

Now expected outcome for validation

s3.isfile("/bucket/inner")  # returns True
s3.isdir("/bucket/inner")  # return True
s3.isfile("/bucket/inner/")  # returns False
s3.isdir("/bucket/inner/")  # returns True
s3.isfile("/bucker/inner/stuff")  # returns True

The most challenging part is the deletion

s3.rm("/bucker/inner")  # should delete the file key
s3.rmdir("/bucket/inner")  # should raise an exception that directory is not empty
s3.rmdir("/bucker/inner", recursive=True)  # should delete both keys "/bucker/inner/" and "/bucker/inner/stuff"

> a key exists at "/bucket/inner/" which contains data and/or metadata

boto3 allows storing data for the "/bucket/inner/" key. Check the following output (highlighted directory content size)

% aws s3 ls s3://bucket/
PRE inner/
2021-01-07 19:00:01          0 inner
% aws s3 ls s3://bucket/inner/
2021-01-07 18:58:05          0
2021-01-07 18:57:38          0 stuff
% aws s3 ls s3://bucket/inner/

boto3_s3_client.put_object(Bucket='bucket', Key='inner1/', Body="sdf")
boto3_s3_client.put_object(Bucket='bucket', Key='inner1/stuff', Body="sdf")

% aws s3 ls s3://bucket/
PRE inner1/
% aws s3 ls s3://bucket/inner1/
2021-01-07 19:28:11          3
2021-01-07 19:28:47          3 stuff

> You could solve the issue by keeping track of mkdirs within the instance, not touching the remote storage backend.

I would not make an assumption for end-user of the s3fs library. s3fs is just a wrapper library end of the day. And documenting that mkdir is a no-op breaks that rule of the wrapper library responsibility

Thank you for thinking through some of the scenarios, and I hope you realise that there are more! The point is, whether you still to posix-on-s3 (requiring directories) or pure-s3 (the current), you expose holes in the mismatch and have to hope that users see things the same way you do.

boto3 allows storing data for the "/bucket/inner/" key

Yes, but is this a file, directory, or both?
The final  and perhaps most compelling case for leaving things how they are, is that if mkdir is taken to mean "make the namespace below this writable" (as opposed to "create a directly-like object"), then the current implementation does this without a call to s3, which may be important. Libraries like parquet IO might call mkdir a lot, and this can hinder performance.

Thank you for thinking through some of the scenarios, and I hope you realise that there are more! The point is, whether you still to posix-on-s3 (requiring directories) or pure-s3 (the current), you expose holes in the mismatch and have to hope that users see things the same way you do.

boto3 allows storing data for the "/bucket/inner/" key

Yes, but is this a file, directory, or both?
The final  and perhaps most compelling case for leaving things how they are, is that if mkdir is taken to mean "make the namespace below this writable" (as opposed to "create a directly-like object"), then the current implementation does this without a call to s3, which may be important. Libraries like parquet IO might call mkdir a lot, and this can hinder performance.

So if the s3fs project's prime goal is to cater for high performance for libraries like parquet IO. In that case, it's better to keep it the way it is.
Ideally, there should be two versions of this project or distinguished by the constructor parameter (normal vs light version). That way for small projects where the expectation is to have a directory as the real object in s3, then normal version code can be used. Or for namesake call it anything distinguishing it from the current high-performance version.

So if the s3fs project's prime goal is to cater for high performance for libraries like parquet IO. In that case, it's better to keep it the way it is.
Ideally, there should be two versions of this project or distinguished by the constructor parameter (normal vs light version). That way for small projects where the expectation is to have a directory as the real object in s3, then normal version code can be used. Or for namesake call it anything distinguishing it from the current high-performance version.

I agree with this, it would be nice to have the option of "console-style" behaviour with directory objects, even if it's not the default. I'm not sure it's easier to implement, though, since you still have to care about buckets that don't follow that convention.

I agree with this, it would be nice to have the option of "console-style" behaviour with directory objects, even if it's not the default. I'm not sure it's easier to implement, though, since you still have to care about buckets that don't follow that convention.

s3 does not support directories, except the top-level buckets, which are fairly similar in their behaviour. If you expect s3.makedirs('my-bucket-name/testdir') to enable you to write files with paths that start with that string then: congratulations! There is in fact nothing for s3fs to do, the location was already writable.

@martindurant Would it make sense to either disable or raise a warning for mkdir, mkdirs, makedir, and makedirs? The takeaway for me from this issue is that these don't really do anything because of S3's object-store paradigm.

That would make sense if mkdir were only called by end users, but other libraries call it internally. Since the intent there is "make sure that files can be written as children of this path", the co-op does exactly the right thing - and errors and warnings would just get in the way.

@martindurant
The silent creation of the key without an "apparent" directory causes fsspec to fail in certain cases as well.
Not sure where would be the best way to fix this? Maybe the check, create kwargs are redundant in fsspec or, we need to change something here in s3fs.
The check kwarg for fsspec.FSMap expects the "directory" key to exist.
There is a create kwarg to make it "create" that folder, so check can run in cases where it doesn't exist.
Basically, for a root = 'my_bucket/dir_i_want/', if it doesn't exists we can pass create=True and it runs a s3fs.core.S3FileSystem.mkdir(root) call. However, when queried by s3fs.core.S3FileSystem.exists(root), just couple lines later, it still returns False. Hence it fails and tells us to pass create=True.
This is all related to trying to debug weird behavior I've seen at zarr-developers/zarr-python/#814

Possibly it's the exists call in FSMap that's not required. We nest test whether we can write something, that should be enough; or for read-only (which is not explicitly labeled anywhere) that we can list the resultant map.

I agree, it sounds like the check and create in fsspec are redundant. I will open an issue there to discuss further.

Howdy !
In my project I have a somewhat complex application that must transparently work over a local folder or a S3 bucket. One of the wayt ot acheive this was to use  s3fs-fuse to mount a local file through the OS. However, I had some limitations running s3fs-fuse in an OpenShift container so I'm been looking at a pure python layer.
Yet, S3FileSystem is quite limited to perfectly emulate the real file system.

S3FileSystem inherits from fsspec.AsyncFileSystemand therefore is it advertising itself as a file system, which means folder support must be there. Yet, it is not and as such anyone working with S3FileSystem needs a lot of boilerplate to fix folder support.
s3fs-fuse creates keys in the bucket ending with a slash / to signify it is a folder https://github.com/s3fs-fuse/s3fs-fuse/blob/f8a825e9b969a2e4cae89f86a4a47d7458ea7e81/src/s3fs.cpp#L1078
I tried to fake that with S3FileSystem by appending a "/." or "/.folder" to create a stray file. However, I tried to monkeypatch a bit the code in S3FileSystem not to strip trailing slashes and the key was created and then recognized as s3fs-fuse as a folder.

So, I suggest S3Filesystem finally uses "path/" as a format to keep folders and thus improve compatibility with regular file systems which implement fsspec.AsyncFileSystem. This would apply to everything folder related.

The problem you are facing is not really with s3fs, but with the fuse integration. Indeed, s3 does not have directories. You can do things to try to get around that (such as the zero-length files), but these are conventions circumventing the actual storage model and API. When using S3FileSystem directly, mkdir() always succeeds, in that you can write to a path within a directory (you always can).
The problem arise in trying to cd to an empty directory. But you can write to a path within a directory without first creating or changing to that directory.

The problem you are facing is not really with s3fs, but with the fuse integration. Indeed, s3 does not have directories. You can do things to try to get around that (such as the zero-length files), but these are conventions circumventing the actual storage model and API. When using S3FileSystem directly, mkdir() always succeeds, in that you can write to a path within a directory (you always can).

Well, I think we all know that. But the problem is that S3FileSystem tells the world it is a file system and implements the fsspec interface, while actually not implementing file system operations with folders. So, this breaks the contract.

It does the best that it can, given the limitations of the actual backend storage. mkdir() does work in the most important sense of making a path in which you can write files.Having a side effect of creating files seems like a bad idea and even less filesystem-like.

An issue I encountered with this is when combining simplecache and a blob-style backend (in my case, abfs) with a pandas to_parquet. I can't create a directory because mkdir just passes (no directories in blobs), but the to_parquet fails because the directory does not exist in the case. Any suggestions on how to solve this?

If you really want to, you can
fs.touch("bucket/path/")
# or
fs.touch("bucket/path/content")

This is not a directory, but depending on what's doing the checking, may be good enough for you.

Just to clarify, you can't actually touch a dir like that. Because of an rstrip('/') in the path splitting function, the call to fs.touch("bucket/testdir/") results in:
DEBUG:s3fs:CALL: put_object - () - {'Bucket': 'bucket', 'Key': 'testdir'}

- note the lack of a trailing slash.
My code's directory check is fs.info(path)["type"] == "directory", so this doesn't work. I'm not asking for a fix (I just call put_object myself with the fixed key) unless this behaviour is considered a bug.

Update: just posting my slightly cleaner solution if it may help anyone else. With this, you can touch directories as mentioned above.
class S3FileSystemPatch(S3FileSystem):
    @classmethod
    def _strip_protocol(cls, path):
        # don't want to strip '/' off the end of folders, as amazon needs this
        # to create "folders" in the console, as well as for them to appear as
        # directories in file listings
        ret = super()._strip_protocol(path)

        if isinstance(ret, list):
            for i, (src, dst) in enumerate(zip(path, ret)):
                if src.endswith("/") and not dst.endswith("/"):
                    ret[i] += "/"
        else:
            if path.endswith("/") and not ret.endswith("/"):
                ret += "/"

        return ret

... and then use S3FileSystemPatch instead of S3FileSystem


No sync to Sensor / Merker values in Loxone

Hi
I really like your great integration, but I have a problem where I do not how to solve it:
I'm trying to set the value, in detail the temperature of a Hue Motion sensor, of a Loxone Merker (i tried a Virtual Input as well) in HA. Basicly its very straight forward and I can indeed set the correct value of the entity. But the value never gets synced to Loxone and got reset to the value of Loxone after a few seconds
I tried both ways:

set in an automation the temperature value with the entity_id
use the websocket direct command with the correct uuid

Am I missing something?
Because i can easily set the Lights on/off and other stuff with your integration, but I cannot sync as it seems a numeric value to Loxone.
Thanks for your help

Okay, this one is closed now
I tried about 4 days to get it running, and surprisingly it just works by now.


Fix many untagged true/false/null in the documentation

This PR fixes as many instances of true, false, and null not surrounded by [code] tags as I could manage to find.
For the descriptions affected, a few tweaks have been made to justify a whole translation string change. At least, attempted to. Some of these descriptions are not exactly top-notch, and a focused overhaul would make the surrounding descriptions inconsistent in comparison. So, I would rather not.

Updated to include more of these.

With your suggestions to change "Returns" on both of those Audio classes, it actually becomes inconsistent with the surrounding descriptions. I changed them anyway, but it's worth keeping in mind.

Will do a sweep of the use of "e.g." I think, it's far more obscure IMO than "i.e." so I think it'd be worth it to change to "for example" where it is used as such and fix potential cases of using it instead of "i.e."

With your suggestions to change "Returns" on both of those Audio classes, it actually becomes inconsistent with the surrounding descriptions. I changed them anyway, but it's worth keeping in mind.

I think this is the kind of change that should always be made when making other changes, even if it leaves nearby sections a bit mismatched. Because of the implicit "This method..." at the start of each method description, then "Returns foo" is just correct and "Return foo." is just incorrect. Compare:

[This method] returns foo.
[This method] return foo.

"Return foo" does read fine as-is, so it's not worth going out of the way to change, even if it's ungrammatical if you think more about it.

Found one more in GLTF.import_as_skeleton_bones() and I could not let it go.

Oh shoot, did I somehow merge the prior version of this PR??

That was you that did with very precise timing.

Uhm, what now then?

I guess I'll just merge it for realsies the next batch
In any case, we now have a damn good reason to look into setting up a merge queue lmao

I guess I'll just merge it for realsies the next batch

I'm surprised there isn't a gazillion of merge conflicts if a previous version of this commit has been merged already. When this happens, I would recommend to close the PR as indeed merged, and make further changes in a follow-up PR. Merging the same set of changes twice seems pretty weird to me, I'm not sure how Git deals with it and why it considers it non conflicting.

Right, I'll open a PR just for the leftover


chore(SCT-1262): adding two new columns for elements handling

Link to JIRA ticket
https://hackney.atlassian.net/browse/SCT-1262
What changes have we introduced
Alter the table currently used to store case status fields in order to store the element type and where in the lifecycle it appears.
Checklist

[ ] Added end-to-end (i.e. integration) tests where necessary e.g to test complete functionality of a newly added feature
[ ] Added tests to cover all new production code
[x] Added comments to the README or updated relevant documentation (add link to documentation), where necessary.
[ ] Checked all code for possible refactoring
[ ] Code pipeline builds correctly

Closing as the DB structure has changed and we don't need this anymore.


How can I set the title of VirtualDocument?

In virtual-document-sample it seems that the title of the tab is equal to the given text, instead of being the whole text of the file. I am unable to find the line in the example that sets a title. How is it done? In my extension it displays the whole content on hover and a cut-off version by default.

I want this:
https://github.com/Microsoft/vscode-extension-samples/raw/main/virtual-document-sample/cowsay.gif
But have this:

The path or the uri of a virtual document defines its name, so someScheme://path/with/name.txt will use name.txt as name


gpg: troubleshooting fetch

Yanked it out of another branch. This is just the fetch changes that are refusing to work. Everything else is back in the other branch or syntax PR still.
Mostly just trying to work out why after trying to do this a bunch of different custom ways locally, moving to the patch method because we use it already, it works well & seems sane is completely broken for me.

Closing this as we trouble-shooted it.


Library is not unloaded

During my analysis of the framework I noticed that unloading of libraries is not performed.
Here I took your example as a reference and load the library via the framework and start it.

The DLL is successfully loaded by the method 'std::exception_ptr BundlePrivate::Start0()'.

  if (useActivator) {
    try {
      void* libHandle = nullptr;
      if ((lib.GetFilePath() == util::GetExecutablePath())) {
        libHandle = BundleUtils::GetExecutableHandle();
      } else {
        if (!lib.IsLoaded()) {
          lib.Load(coreCtx->libraryLoadOptions);
        }
        libHandle = lib.GetHandle();
      }
...

When I stop or even uninstall the library, the loaded library is not unloaded.
Therefore the use case is not given that one can stop the DLLs and update and start afterwards. Since an active file handle remains active via the process.
The SharedLibrary class is a member of the bundle instance, but it is only used for loading the library. For the unloading the Unload method is not used at all and thus the FileHandle would remain eternally open as long as the instance is executed.
For example, I forced the uninstall operation in the Uninstall operation and was able to successfully unload the DLL.
void Bundle::Uninstall()
{
  if (!d) {
    throw std::invalid_argument("invalid bundle");
  }

  d->Uninstall();
  // Extend this code to unload the lib
  auto private_bundle = GetPrivate(d);
  private_bundle->lib.Unload();
}

Is this currently an error because the DLL was forgotten to be unloaded?
For which operation should the DLL be unloaded Stop and Uninstall or only Uninstall?
With a destructor call this operation of the DLL unloading would have to take place in any case.

The reasons for this are understandable.
The problem that arises is that the file still carries active handles and one could not replace them. Thus a shutdown of all processes would be necessary so that all handles are given up. From my point of view also a clean thing for the update management. Since one receives with the framework a "simple" possibility to terminate all processes properly.
In any case, it is a pity that you can not exchange the small dlls, because this would make the framework even more elegant.
Thanks for the quick answer, this is enough for me.


ungoogled-chromium: 129.0.6668.100-1 -> 130.0.6723.58-1

https://chromereleases.googleblog.com/2024/10/stable-channel-update-for-desktop_15.html
This update includes 17 security fixes.
CVEs:
CVE-2024-9954 CVE-2024-9955 CVE-2024-9956 CVE-2024-9957 CVE-2024-9958
CVE-2024-9959 CVE-2024-9960 CVE-2024-9961 CVE-2024-9962 CVE-2024-9963
CVE-2024-9964 CVE-2024-9965 CVE-2024-9966
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 350073 run on x86_64-linux 1

3 packages built:

ungoogled-chromium
ungoogled-chromium.sandbox
vscode-extensions.yzane.markdown-pdf


Document by shortuct for groupBy

We've implemented shortcut for by argument of groupBy queries in prisma/prisma#20296. Now, if you want to group by a single field you don't need an array.
We need to document it. groupBy is documented on 2 pages:

https://www.prisma.io/docs/concepts/components/prisma-client/aggregation-grouping-summarizing#group-by
https://www.prisma.io/docs/reference/api-reference/prisma-client-reference#groupby

Related to https://github.com/prisma/prisma/issues/20261


Rename master/slave to controller/worker in the testing infrastructure.

This is just to avoid the controversial pair of words "master/slave". The change aligns with how those things were named in Scala Native.
The method sbt.testing.Framework.slaveRunner is not renamed, because doing so would break binary compatibility.
In BinaryIncompatibilities, we merge TestCommon and TestAdapter. TestCommon used to be reused for the test-bridge artifact, but that one does not have any binary compatibility constraints anymore, so the distinction was artificial.

This PR is a direct consequence of the Scala Center Advisory Board Recommendation SCP-025, which was adopted at the latest Advisory Board Meeting. Although we have not worked on a list of words to avoid yet, it is clear that the "master/slave" association will end up on the eventual list (even if it's the only thing that ends up there). So this PR is a first step in that direction.
/cc @olafurpg, since you were interested in the issue with the slaveRunner method.

LG overall, just one question: Does "controller" accurately convey that this Runner instance is expected to run tests as well? Is there an industry standard about this? Otherwise maybe "main" would be a better name ("primary" also comes to mind, but this IMO implies ordered-ness to a larger extent than a single out of many).

There's a bunch of alternatives at https://en.wikipedia.org/wiki/Master/slave_(technology)#Replacements, but none truly convinces me. Anything that says "replica" is plain wrong. I did hesitate with leader/follower, but IMO it doesn't convey the controlling aspect of the leader enough. At least "controller" does not contradict the fact that it's working as well. I'm not aware of any "industry standard" terminology that says "I'm giving orders but I'm working too.".

Also, it's worth pointing out that controller/worker was already accepted a few months ago in the Scala Native repo. Unless there is strong concern against that naming, we should keep it. Otherwise we shuld get other Scala Native contributors on board with another rename, because ultimately we should have the same names in both repos.

It looks like Spark uses Driver/Worker but after reviewing the list on Wiki, I think that controller/worker is about the best combo.

The Spark Driver doesn't do any work though IIUC.

Let's wait until the Scala Center has produced the actual guidelines before committing to the new names.

A few moons later ... the Scala Center's guide for inclusive language is published: https://www.scala-lang.org/contribute/inclusive-language/
So I rebased this PR.

Coming back to this I think Main/Worker would a better name, but we should probably just stick with Controller/Worker for the following reasons:

It's already in Scala Native (I'm not really happy about this precedent though; in my understanding, Scala Native is the less stable repository of these two, so if we let it drive precedent cases like this, it might drive code quality down)
It seems to be a standard pair (at least in the Scala Center's guide).
It's not that important :)

Honestly, the more I think about this, the less I think it matters that much. In the end, the precise interaction semantics will never be entirely captured by a name.
The problem with "Main/Worker" is that the words don't form an obvious pair (compare with Primary/Secondary)...

OK. Let's keep what we have here, then, and stop worrying about it. 😉


React-Flows scrolls to inifinity with Remix

Describe the Bug
When used with Remix the react-flow "map" scrolls off down the page to infinity by itself.
I was trying to solve the issue on the Remix discord channel and a member provided an example on twitter that has the same issue.
The odd thing is that react-flow works fine in incognito mode yet not in normal mode in Chrome.
The sample:
https://twitter.com/kiliman/status/1555175514167558144
It has been suggested it might be a hydration issue with React and react-flow.
Unfortunatley this would not lead to a great customer experience if they had to use incognito to use the application so I am hoping someone might know the cause.
Thanks in advance :)
Your Example Website or App
https://twitter.com/kiliman/status/1555175514167558144
Steps to Reproduce the Bug or Issue

Go to the CodeSandbox example
Notice on the right side where it is rendered the react-flow map disappears off the screen by itself.

NOTE:
This also happens if you create a brand new Remix app locally and add react-flow
Expected behavior
I expect the same output in normal mode as incognito mode.
Screenshots or Videos
No response
Platform

OS: Windows 10
Browser: Chrome
Version: Version 103.0.5060.134

Additional context
It has been suggested that a chrome extension is causing the hydration issue but I am not sure if that would be the case - plus we don't want to have to tell customers to not use extensions in Chome :/
Thanks team!

The codesandbox and also a local Remix app work fine for me. Could you provide any more information here?

Please re-open if the issue still persists. I can't reproduce this.


patch: add support for changing directory

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

[ ] Have you successfully run brew tests with your changes locally?

Example: Homebrew/homebrew-core#51129 could be rewritten as
patch do
  url "https://github.com/boostorg/boost_install/compare/52ab9149544bae82e54f600303f5d6d1dda9a4f5...a1b5a477470ff9dc2e00f30be4ec4285583b33b6.patch?full_index=1"
  sha256 "fb168dd2ddfa20983b565ead86d4355c6d6e3e49bce9c2c6ab7f6e9cd9350bd4"
  directory "tools/boost_install"
end

Btw, you can apply a patch for a resource by adding the patch block to resource block, for example https://github.com/Homebrew/homebrew-core/blob/eb246b43ff3fec92fbdeaa615db5f988cf87cf50/Formula/azure-cli.rb#L486-L495
If this is possible to do it looks more preferable for me since it keeps related things together (especially useful if you have a lot of resources)

Ah, I misread the example, my comment above completely unrelated for this case, sorry for the noise 🤷♂

Yeah, the Boost example is perhaps a tad confusing - which is probably a good reason to incorporate proper patch subdirectory support!

Done.

@Bo98 GitHub Actions being weird; can you force push this? Ta!

Nice work again @Bo98!


Laravel 9 Compatability

Changed the composer.json, the illuminate/support version to a min of 9.0, and PHP, removing the 7.4 dependency. So far so good. (Readme changes were an error and corrected).

Thanks @PeachySoftware — We're doing an upgrade on one of our projects this month so we'll confirm this update as we go through that.


defenses against CAN attacks

Reviewed, but will not merge until @kentindell's request has been discussed

agreed @kentindell , @jmaag15 . I've collected the CAN protocol attacks together into our model of a composite requirement in the latest

janus attack requirement now reflects group concensus 20220428 that janus attack should be expected on UND but not on TND

I'm good with those changes. One final question: the requirements as stated are for Janus and Address Claim attacks on UND. Does this mean:

That the gateway should detect and not pass through Janus frames, and
That the gateway should detect and frustrate the address claim attack?

(1) is a kind of IDS function. It's OK to require this, and I think it's implementable on a sufficiently good microcontroller.
(2) raises some issues. Right now, is the address claim protocol required to work through the security gateway? I don't understand it in enough detail yet but AIUI it exploits the CAN error passive mechanism to do a meta-arbitration. That means that the security gateway would have to try to replicate this across two separate buses, which is not just tricky (because there's a sync problem keeping both sides together) but also could constitute a flood attack on TND, stimulated by a device on UND.

janus attack requirement now reflects group concensus 20220428 that janus attack should be expected on UND but not on TND

Yep, I'm good with this. When the CVE for the Janus attack is officially released (which is any day now) then maybe it's worth taking some of the mitigations in that bulletin into the ECU hardening requirements (like locking down the pin mux). That would then underpin this UND-only requirement.

@kentindell

2. That the gateway should detect and frustrate the address claim attack?

I think
[REQUIREMENT]
UID: J1939GW-S-029
REFS:
- TYPE: Parent
  VALUE: AGW-S-002
TITLE: Prevents Address Claim Attacks
STATEMENT: The device SHALL prevent generating address claim attacks on *TND* from messages originating on *UND*. c.f. https://www.aut.upt.ro/~bgroza/Papers/j1939.pdf and https://doi.org/10.4271/02-14-03-0026 for a description of the address claim attack.

isn't in turn requiring detection or an active defense. e.g. if there is no address claim gatewaying then the requirement is trivially satisfied. In the case where gatewaying of address claim is a feature then tracking, rate limiting or something else is needed to prevent the attack on TND from UND.

Just one comment: "expected to be impervious to Janus Attacks since it should all be OEM supplied components there" holds only if there are mitigations to prevent an OEM device being hijacked (e.g. by diagnostic stack bug) and then carrying out a Janus (or other CAN protocol) attack on other devices.
Simple mitigations for this should be in place: sample points verified as confirming to J1939 (to defend against Janus), enabling of pin mux lock by secure bootloader or use of an external CAN controller (to mitigate carrying out of CAN protocol attacks).

Just one comment: "expected to be impervious to Janus Attacks since it should all be OEM supplied components there" holds only if there are mitigations to prevent an OEM device being hijacked (e.g. by diagnostic stack bug) and then carrying out a Janus (or other CAN protocol) attack on other devices.

Not sure I agree, @kentindell. If all devices are impervious to reception of Janus frames (no devices will differ on their deserialization of any message sent) then even if a TND device is compromised and a Janus protocol level bitstream is sent the attack will not succeed because all nodes receive the same 'side' of the janus bitstream.

Not sure I agree, @kentindell. If all devices are impervious to reception of Janus frames (no devices will differ on their deserialization of any message sent) then

Correct. But that requires that the ECUs are actually set up to be impervious. J1939 does specify a single sample point, but I can see how if it's not an overt security requirement to do so then it can get missed out: it's unlikely to show up in testing if someone just used 50% as the sample point.
I don't think this is a requirement for security gateways and it's out of scope here, but it should exist in an ECU security requirement document somewhere.

Correct. But that requires that the ECUs are actually set up to be impervious.

Yep -- that's my point :) OK we're in violent agreement here 💯

Reviewed, reflects discussion on 20220505, merging


[First party package suggestion] Dart needs a native webp encoder

I know this is the SDK repo, but I don't know where to raise this.
Dart is sorely missing native webp encoding. Image is the most popular package and the maintainer doesn't have the time to work on it. Perhaps the Dart team need to create a first party image encoding package? It's a really hassle having to shoehorn other languages & tech in to do this task in my otherwise exclusively dart tech stack.
(If it were a package I could make myself I would!)
Thanks,

@jimmyff Have you found any alternatives?


The numbe in Scatter Chart is not same as Response Summary

Why the number called success of Scatter Chart is not equals to Response Summary ? The Error num in Scatter Chart is not same as Response Summary either , why?

Hi @Eilikce
The numbers between the scatter chart and response summary may be different due to the reasons explained in #2381. The response summary chart numbers work the same way as the server map numbers so they may be different from the scatter chart numbers.
The "Error" number above the scatter chart is the number of agents that had an error.
So in your case, Total 1        Error 1 means there is 1 agent registered under the application name, of which 1 agent had a failed transaction.

Why pinpoint use different time wides in these charts ? The charts usually can be see in a common web page . Is better that use same time wides ?

The scatter chart data on the other hand have 1 second granularity, so these can be differentiated in 1 second interval.
The server map, response summary, and load chart data on the other hand, are stored in 1 minute granularity (the collector aggregates these in memory and flushes them every minute due to performance reasons).
So the only way to make everything use the same time intervals would be to query everything in 1 minute granularity. This means if you query for 10:15:30 ~ 10:20:30 data, the scatter chart will query everything from 10:15:00 ~ 10:21:00 and display these. This would probably make all the numbers sync up, but then you lose the ability to easily select transactions that happened before 10:20:30 (you'll have to carefully drag on the scatter chart).
So in the end, you've got to decide what's more important - make the numbers match, or provide a way to filter out the scatter chart more accurately depending on the query time.
We chose the latter.


How to compute the ERes2Net model param?

Hello, I use the same model params as your configs in https://github.com/alibaba-damo-academy/3D-Speaker/blob/6f6ed3189a4d1db040586a518c8e5d80f4fc0665/egs/3dspeaker/sv-eres2net/conf/eres2net.yaml, but I get 9.88M. (Yours is 4.6M)
Here is the way I compute the model params:

I'm wondering where the difference is ?

Even I set embedding_size=192, I still got 6.61 M.

The key difference arises from how we compute the model parameters. Since a classifier isn't used during inference, it's not factored into the statistical calculation of the model parameters.

Thank you for your answer!
But what part of the ERes2Net model is the classifier you mean? Is it the output linear layer mapping the embedding into the speaker label? However, it is not defined in the model.

Yes, the classifier refers to the output linear layer mapping the embedding into the speaker label. Therefore, since these parameters will be discarded during inference, they are not factored into the model parameter calculations.

Thank you for your answer. I directly initialize the ERes2Net model as defined in ResNet.py, which does not contain the classifier as you mention above. The code lines in the screenshot are directly appended in the end of your ResNet.py and run python ResNet.py. So I think my calculation result should be consistent with yours. What is wrong with my codes?
It would be nice if you can share the codes you calculate the model parameters. Thanks so much!

Apologies for my oversight, I overlooked the parameters following the statistical pooling layer. With an embedding size of 192, the model parameters total 6.61M. When the embedding size is 512, the model parameters amount to 9.88M. I'll update this on arXiv paper and GitHub soon. Thank you very much for reminding.


Use @javax.ws.rs.Path value as transaction name

…@Path annotation.
closes elastic/apm-agent-java#346

The annotations may even not at all be present on the resource class but rather on a superclass or an implemented interface.

@felixbarny I added method to determining the value of annotation in the case of a superclass or implemented interfaces.

What about path annotations on methods? Those need to be appended to the class @Path value, if such exists, as they are way to sub-resource the main resource.
I think they may be used even without the class being annotated with @Path, meaning that we need to add them to getMethodMatcher.
@felixbarny @kananindzya WDYT?
We can start by adding such test cases and see how they behave.
Another test case to add is for path params, as in
@Path("employees")
 public class EmployeeResource {
  @GET
  String getList() {...}

  @GET @Path("{id}")
  String getEmployee(@PathParam("id") String id) {...}
 }

@eyalkoren
How I understand, here need to extend SimpleMethodSignatureOffsetMappingFactory ?
If CoreConfiguration.isUseAnnotationValueForTransactionName() enabled, then we should look for an annotation of the type javax.ws.rs.GET(POST, ...), and Path?

@felixbarny, right now AnnotationValueOffsetMappingFactory turned out to be a complex object.
And I need to add value from method Path annotations.

Ideally, it would look like this: GET /testWithPathMethodSlash/{id}
On Wed 12. Jun 2019 at 12:58, kananindzya<EMAIL_ADDRESS>wrote:

@kananindzya commented on this pull request.
In
apm-agent-plugins/apm-jaxrs-plugin/src/test/java/co/elastic/apm/agent/jaxrs/JaxRsTransactionNameInstrumentationTest.java
https://github.com/elastic/apm-agent-java/pull/620#discussion_r292855194
:

@@ -205,6 +263,21 @@ public String testMethod() {
}
}

@Path("testWithPathMethod")
public static class ResourceWithPathOnMethod extends AbstractResourceClassWithoutPath {

   @Override

   public String testMethod() {

       return "ok";

   }

   @GET

   @Path("{id}")

@felixbarny https://github.com/felixbarny,
Should it look like this or not
GET /testWithPathMethodSlash//{id}
?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/elastic/apm-agent-java/pull/620?email_source=notifications&email_token=AAQQGCGJR5CX2S42NQXJJN3P2DJHDA5CNFSM4HJ5KYP2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOB3JKHHQ#discussion_r292855194,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAQQGCHBLILHSXBJW4B3CK3P2DJHDANCNFSM4HJ5KYPQ
.

I have added some suggested changes here: https://github.com/kananindzya/apm-agent-java/pull/1
As soon that is in, I think we are ready to merge!

@kananindzya very few stuff left to finalize this great PR. Let me know if you don't want to handle them and prefer I take over.

@kananindzya very few stuff left to finalize this great PR. Let me know if you don't want to handle them and prefer I take over.
Hi @eyalkoren , yes, I’ll finish it soon, sorry for the delay, have been very busy lately.

@kananindzya very few stuff left to finalize this great PR. Let me know if you don't want to handle them and prefer I take over.
Hi @eyalkoren , yes, I’ll finish it soon, sorry for the delay, have been very busy lately.

@eyalkoren, please verify last changes.

@kananindzya just address https://github.com/elastic/apm-agent-java/pull/620#discussion_r299403588 and https://github.com/elastic/apm-agent-java/pull/620#discussion_r299409993 and let me know

@kananindzya just address #620 (comment) and #620 (comment) and let me know

fixed it, in addition added variable useAnnotationValueForTransactionName in JaxRsOffsetMappingFactory.

Sorry, this is not what I meant...
The way it was before (passing the tracer in the constructor) is better, you can revert to that.
What I meant is to change from:
    public Advice.OffsetMapping.Factory<?> getOffsetMaping() {
        return new JaxRsOffsetMappingFactory(elasticApmTracer);
    }

to
    public Advice.OffsetMapping.Factory<?> getOffsetMaping() {
        return new JaxRsOffsetMappingFactory(tracer);
    }

and remove the field.
The ElasticApmInstrumentation#tracer field is initiated before this method is called

Sorry, this is not what I meant...
The way it was before (passing the tracer in the constructor) is better, you can revert to that.
What I meant is to change from:
    public Advice.OffsetMapping.Factory<?> getOffsetMaping() {
        return new JaxRsOffsetMappingFactory(elasticApmTracer);
    }

to
    public Advice.OffsetMapping.Factory<?> getOffsetMaping() {
        return new JaxRsOffsetMappingFactory(tracer);
    }

and remove the field.
The ElasticApmInstrumentation#tracer field is initiated before this method is called

reverted

@kananindzya one last thing - you need to run mvn test so that the new config option will be added to the documentation and then push the resulted configuration.asciidoc file

@kananindzya one last thing - you need to run mvn test so that the new config option will be added to the documentation and then push the resulted configuration.asciidoc file

updated

@kananindzya Since I promised the last one would be last, I did the last one myself - https://github.com/kananindzya/apm-agent-java/pull/2/files 😄
I just realized the new config option needs to be moved to the JAX-RS category.
Just merge this and we're really done..

@kananindzya Since I promised the last one would be last, I did the last one myself - kananindzya#2 smile
I just realized the new config option needs to be moved to the JAX-RS category.
Just merge this and we're really done..

Done :)

Jenkins run the tests

Codecov Report

Merging #620 into master will increase coverage by 1.94%.
The diff coverage is 54.52%.

@@             Coverage Diff              @@
##             master     #620      +/-   ##
============================================
+ Coverage     62.87%   64.81%   +1.94%
  Complexity       68       68
============================================
  Files           192      204      +12
  Lines          7684     8523     +839
  Branches        949     1114     +165
============================================
+ Hits           4831     5524     +693
- Misses         2554     2692     +138
- Partials        299      307       +8

Impacted Files
Coverage Δ
Complexity Δ

...tic/apm/agent/configuration/CoreConfiguration.java
97.71% <ø> (+0.06%)
0 <0> (ø)
:arrow_down:

...m/agent/jms/JmsMessageListenerInstrumentation.java
0% <0%> (ø)
0 <0> (?)

...m/agent/jms/JmsMessageProducerInstrumentation.java
0% <0%> (ø)
0 <0> (?)

.../elastic/apm/agent/jdbc/helper/JdbcHelperImpl.java
0% <0%> (ø)
0 <0> (ø)
:arrow_down:

...astic/apm/agent/jdbc/StatementInstrumentation.java
50% <0%> (-1.73%)
0 <0> (ø)

.../elastic/apm/agent/jms/BaseJmsInstrumentation.java
0% <0%> (ø)
0 <0> (?)

...java/co/elastic/apm/attach/ElasticApmAttacher.java
0% <0%> (ø)
0 <0> (ø)
:arrow_down:

...nt/httpclient/HttpAsyncRequestProducerWrapper.java
0% <0%> (ø)
0 <0> (ø)
:arrow_down:

...ain/java/co/elastic/apm/attach/RemoteAttacher.java
32.01% <0%> (-0.49%)
0 <0> (ø)

...lastic/apm/agent/jms/JmsInstrumentationHelper.java
0% <0%> (ø)
0 <0> (?)

... and 55 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 305e177...3e1d329. Read the comment docs.


Update 'Juices' image on Cafe page

Please refer to my email for image.

This is done per my email


Fix wind downscaling under Iris 2.1

On our latest Python 3 stack (Iris 2.1, Python 3.6.6, Dask 0.18.1) we have an exception when running improver wind-downscaling:
Traceback (most recent call last):
  File "/opt/improver/bin/improver-wind-downscaling", line 156, in <module>
    main()
  File "/opt/improver/bin/improver-wind-downscaling", line 133, in main
    wind_speed = wind_speed_list.merge_cube()
  File "/lib/python3.6/site-packages/iris/cube.py", line 382, in merge_cube
    merged_cube, = proto_cube.merge()
  File "/lib/python3.6/site-packages/iris/_merge.py", line 1241, in merge
    merged_data = as_concrete_data(merged_data)
  File "/lib/python3.6/site-packages/iris/_lazy_data.py", line 156, in as_concrete_data
    data, = _co_realise_lazy_arrays([data])
  File "/lib/python3.6/site-packages/iris/_lazy_data.py", line 120, in _co_realise_lazy_arrays
    computed_arrays = da.compute(*arrays)
  File "/lib/python3.6/site-packages/dask/base.py", line 402, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/lib/python3.6/site-packages/dask/threaded.py", line 75, in get
    pack_exception=pack_exception, **kwargs)
  File "/lib/python3.6/site-packages/dask/local.py", line 521, in get_async
    raise_exception(exc, tb)
  File "/lib/python3.6/site-packages/dask/compatibility.py", line 69, in reraise
    raise exc
  File "/lib/python3.6/site-packages/dask/local.py", line 290, in execute_task
    result = _execute_task(task, data)
  File "/lib/python3.6/site-packages/dask/local.py", line 270, in _execute_task
    args2 = [_execute_task(a, cache) for a in args]
  File "/lib/python3.6/site-packages/dask/local.py", line 270, in <listcomp>
    args2 = [_execute_task(a, cache) for a in args]
  File "/lib/python3.6/site-packages/dask/local.py", line 271, in _execute_task
    return func(*args2)
TypeError: op_getitem expected 2 arguments, got 4

If we take a copy of dask, point to that, and hackily alter it to print out what its doing:
--- /lib/python3.6/site-packages/dask/local.py	2018-06-23 15:36:48.000000000 +0000
+++ /opt/improver/lib/dask/local.py	2018-09-12 12:47:09.000000000 +0000
@@ -512,12 +512,18 @@
                 key, res_info, failed = queue_get(queue)
                 if failed:
                     exc, tb = loads(res_info)
-                    if rerun_exceptions_locally:
+                    if rerun_exceptions_locally or True:
                         data = dict((dep, state[cache][dep])
                                     for dep in get_dependencies(dsk, key))
                         task = dsk[key]
+                        print("Tried to run", task, data, key, res_info)
                         _execute_task(task, data)  # Re-execute locally

then we get something like:
Tried to run (<built-in function getitem>, (<built-in function getitem>, array-original-somehexkey, (slice(0, 16, None), slice(0, 970, None), slice(0, 1042, None)), False, False), (None, slice(None, None, None), slice(None, None, None), slice(None, None, None))) {array-original-somehexkey: array([[[9.00655937e+00, 8.97640610e+00, 8.96419144e+00, ...,
[edited out lines]
 1.05287504e+01, 1.05018816e+01, 1.04815016e+01]]], dtype=float32)} (array-stack-someotherhexkey, 0, 0, 0, 0) (TypeError(op_getitem expected 2 arguments, got 4,), <traceback object at 0xaaaaaaa>)

print statements in just above the lines mentioned in the traceback in is_concrete_data and _co_realise_lazy_arrays:
something like:
print("is_concrete_data", data, data.has_lazy_data())

and
print("_co_realise_lazy_arrays", arrays)

gave:
is_concrete_data dask.array<stack, shape=(12, 33, 970, 1042), dtype=float32, chunksize=(1, 16, 970, 1042)> True
_co_realise_lazy_arrays [dask.array<stack, shape=(12, 33, 970, 1042), dtype=float32, chunksize=(1, 16, 970, 1042)>]

Acceptance criteria:
See GitHub ticket.

This problem is hard to reproduce, but can be done by pickling the data pre-merge.
It seems that within merge, the problem is caused by stacking the arrays using dask (making them lazy, then realising them again), it seems to be particular to the dimensionality of the problem, and can only be replicated as a minimal example with similarly large arrays (eg. particularly slices with more than 16 levels), so the problem seems to lie fundamentally with the chunking of the numpy arrays.
Potential solutions:
•	taking smaller sub-slices in either vertical and/or horizontal coordinates (merging twice, rather than once)
•	an alternative to iris.merge that doesn't use dask (namely replacing da.stack with np.stack in iris' multidim_lazy_stack)
•	a manual merge in this case, not using iris' merge, just naively stacking the results and popping out the relevant dim (realization in this case), effectively a concatenate.
None of these are particular attractive solutions, but serve as a sticking plaster until there is an upstream fix (it appears to be fundamentally a dask bug, so could take some time to fix and trickle down into our stack)

Iris chooses an arbitrary number of points (inherited from biggus), over which it reduces the chunksize.
# A magic value, chosen to minimise chunk creation time and chunk processing
# time within dask.
_MAX_CHUNK_SIZE = 8 * 1024 * 1024 * 2

If we increase this number (by a factor of two for our 33 level 970*1042 UK data) we can step around the problem for the time being by stopping the Iris sub-chunking the slices to be merged. As here::
dask.array<array, shape=(33, 970, 1042), dtype=float32, chunksize=(16, 970, 1042)>

As we are using 32bit floats now, for UK data we would still have a chunk size of ~130MB , while global data would expand to ~650MB.

This has now been fixed in https://github.com/dask/dask/issues/4043, we should be able to undo this fix this when the new Dask (>= 0.19.3) hits the stack.