[$500]  IOU - Edit request option is present for paid message and error shows up when editing paid IOU

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 1.3.91-6
Reproducible in staging?: Yes
Reproducible in production?: Yes
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
Issue found when executing PR https://github.com/Expensify/App/pull/29934
Action Performed:

Go to staging.new.expensify.com
Request money from a workspace chat
Click on the expense preview to open expense report
Click Pay elsewhere
Hover over the system message for paid request
Click Edit request
Return to IOU report
Hover over IOU preview and click Edit request
Edit any field

Expected Result:

"Edit request" option should not be available for the system message for paid request
When clicking on "Edit request" on IOU preview, user should be able to edit the IOU details

Actual Result:
"Edit request" option is available for the system message for paid request. Clicking on it does not allow user to edit the request

When clicking on "Edit request" on IOU preview and user edits IOU details, error shows up

Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] Windows: Chrome
[x] MacOS: Desktop

Screenshots/Videos
https://github.com/Expensify/App/assets/78819774/87139de8-a9e0-42e8-ac19-5a28db67891a

Proposal
Please re-state the problem that we are trying to solve in this issue.
IOU - Edit request option is present for paid message and error shows up when editing paid IOU
What is the root cause of that problem?
We're showing edit option for even the system message:
https://github.com/Expensify/App/blob/a67253b715bc30fc5f024d103fedf8f5adb43a54/src/pages/home/report/ContextMenu/ContextMenuActions.js#L283-L284
What changes do you think we should make in order to solve the problem?

I think we shouldn't show the edit option for system message like that, so we need to update the condition here to:
const isSystemMessage = reportAction.actionName === CONST.REPORT.ACTIONS.TYPE.IOU && reportAction.originalMessage.type === CONST.IOU.REPORT_ACTION_TYPE.PAY;

return type === CONTEXT_MENU_TYPES.REPORT_ACTION && ReportUtils.canEditReportAction(reportAction) && !isArchivedRoom && !isChronosReport && !isSystemMessage;

What alternative solutions did you explore? (Optional)
N/A

Will test this and review this by tomorrow.

@mananjadhav did you get a chance to test/review?

@mananjadhav can you give this one a look when you get a chance?

I was stuck with less internet. I'll finish this by tomorrow.

I was able to reproduce and I am evaluating multiple proposals.

Hope you're feeling better! Can you respond to the proposals today? This one has been open for a little while now.

Yes better and I'lll decide on the proposals today. Thanks for the patience.

Taking a look now.

Overall all the proposals identified the root cause, but I prefer @tienifr's proposal as it is looks cleaner.
🎀 👀 🎀 C+ reviewed.

Hmm, will that solution prevent them from editing the money request in general? Or just the system message?
I believe an admin should still be able to edit the request itself, even after payment. But they should never be able to edit a system message.

Ok got it! Did something change? I'm not seeing that option:

@garrettmknight I'm no longer seeing the bug, are you? If not, let's close.

@puneetlath I was able to reproduce it with @tienifr's instructions - When I click the edit button, it doesn't let me edit, but it's there.

I can reproduce it as well!

Let's do it. Sorry for the delay!

Offer sent to @tienifr https://www.upwork.com/nx/wm/offer/100202927

I couldn't pinpoint to an exact PR for this one. Because we have canEditMoneyRequest since long. I also don't think we need a regression test for this one. @puneetlath @garrettmknight wdyt?
@garrettmknight Can you please post a payout summary for me to raise a request?

Summary of payments:

Contributor: @tienifr $500 paid via Upwork
Reviewer: @mananjadhav $500

Dropping to weekly for payment

$500 payment approved for @mananjadhav based on summary above.


Debian package is not extracted recursively

I create a "scan_single_package" project using http://ftp.us.debian.org/debian/pool/main/e/elfutils/elfutils_0.188-2.1_amd64.deb as an input.
The package is not fully extracted recursively, even though the settings are to extract recursively

@pombredanne Could you add the recurse option to the extractcode.api.extract_archive function?

extractcode.api.extract_archive cannot recurse
For this you need to use: extractcode.api.extract_archives instead

@pombredanne The current implementation of extraction for the single package scan is to extract directly the single archive to the codebase work directory.
The extractcode.api.extract_archives extracts in place, so we need to copy the input archive first. This would change the scan root directory of 1 level.
Are you sure you want to apply this change?
Current:

Using extracts:

We could add a new step to run the extract_archives after extracting the single archive to the codebase dir.

Ok, then may be you could keep the extraction the way it is today and add a call to extractcode.api.extract_archives() afterwards to force recursion?

Fixed in https://github.com/nexB/scancode.io/pull/1083


Override template values on the command line

We can now put {varname} variables in our container definitions, and setup default values for those in our task definitions... we also need a way to override these from the command line

Closed by #10


AutoImplement with baseclass

This PR adds an AutoImplement template very much like the existing one, but with the additional feature of deriving from a baseclass while auto-implementing an interface.

Marking this for me to look at later.

...And I managed to forget about it anyway. I'll get to this within the next day or two.

Ping @MetaLang ;-)

Sorry, I did look at it but got sidetracked updating my local repositories so that I can build DMD/Phobos/Druntime and ultimately this PR. TL;DR: I haven't forgotten about this.

Sorry, almost forgot. Can you squash your commits?


Renaming doesn't work when I try to Rename then Move

Hi!
I want all my files' extensions in Downloads folder be converted to lowercase and then moved to different folder depending on its file type.
I'm trying this code on example.PDF file
rules:
  - folders:
      - '~/Downloads'
    filters:
      - Extension
    actions:
      - rename: '{path.stem}.{extension.lower}'

  - folders:
    - '~/Downloads'
    filters:
      - extension:
          - pdf
    actions:
      - move: '~/Documents/'

As I run organize sim it says this file will be renamed and then moved to Documents but as I run organize run it just moves without renaming as example.PDF (not example.pdf)

As for now I've moved rename action into move action
- folders:
    - '~/Downloads'
    filters:
      - extension:
          - pdf
    actions:
      - rename: '{path.stem}.{extension.lower}'
      - move: '~/Documents/'

When you use move action you can rename too. Simple add new file name after / like this:

move: '~/Documents/{path.stem}.{extension.lower}’


Make it work in zoomed context

I'm using this in a "infinite" div which allows zooming using css transfrom scale property. When scaled, the position of the box is in correct because it is positioning based on non-zoomed coordinates. Is there a way to compensate for this?
I could calculate the new coordinates manually on selection change, but I don't know how I can use that to influence the box on selection start. Any tips?

@jamesyu sorry for the late reply. we definitely haven't accounted for this sort of solution. if you come up with something, would you mind sharing with us so we could considering putting it into the package if we think it's appropriate?

I think an option to pass a zoomfactor (e.g. 1.2, 2.0, 3.0) that would move the selection box back to the tip of the cursor, or any other any way to manipulate where the box spawns on the screen would be beneficial.

@sascharissling would you be willing to contribute a PR? It's outside the scope of what we need this plugin for but would be happy to accept a PR!


fix(chat): hide folder checkboxes in Manage attachments (Issue #1665)

Description:
hide folder checkboxes in Manage attachments
Issues:

Issue #1665

Checklist:

[ ] the pull request name complies with Conventional Commits
[ ] the pull request name starts with fix(<scope>):, feat(<scope>):, feature(<scope>):, chore(<scope>):, hotfix(<scope>): or e2e(<scope>):. If contains breaking changes then the pull request name must start with fix(<scope>)!:, feat(<scope>)!:, feature(<scope>)!:, chore(<scope>)!:, hotfix(<scope>)!: or e2e(<scope>)!: where <scope> is name of affected project: chat, chat-e2e, overlay, shared, sandbox-overlay, etc.
[ ] the pull request name ends with (Issue #<TICKET_ID>) (comma-separated list of issues)
[ ] I confirm that do not share any confidential information like API keys or any other secrets and private URLs

/deploy-review

/deploy-review

/deploy-review


Use SQLitePlugin in native code

Is possible to use the SQLitePlugin in an native Android / iOS code to perform queries and sql operations? Is any example of it?
I'm trying to execute a background sync and we want to do the synchronization in background mode, to allow periodic updates and keep the sync running even if the app goes to background.
I think that one possible solution is extends SQLitePlugin and create a custom package, make the method public available and share the sqlite instance between RN and native code (at least in android).
Thanks!

Hi,
I try React Native in last 6 month ago in interactive app for android By sqlite database. I have big problems and try to solve them but at last I Found React Native is so hard to my purposes because my app need to get data from database in many times but handle them and UI was so complex !
I leave react and back to java, I hear some thing about Flutter/ Dart  maybe can help you !!
Good Luck

@AMAISGOD
I think that your comment is personal opinion of RN and don't help in this issue.
Thanks anyway!

that comment was ridiculous so I deleted it. You can try to use the plugin directly as long as you adhere to the API I suppose although there is a LOT of glue code written in JS...


Matrix cache inclusion

Combined speed ups from sigtesting changes and cache inclusion.

: +1 :+1:


Permit a custom options / capabilities payload when instantiating local / remote / bs drivers

Came about from porting site_prism over to using the framework

This is now implemented. Just needs to be cut as a v5.1


fix(QueryCache): Use default configuration used for QueryCache instance for context

Context
See test cases but when using useQuery with no config provider and a custom query cache provider, the defaultConfig was not being merged correctly in queryCache.buildQuery.
A previously failing test:
test('uses defaultConfig for queries when they don\'t provide their own config', async () => {
    const cache = makeQueryCache({
      defaultConfig: {
        queries: {
          staleTime: Infinity,
        },
      },
    })

    function Page() {
      const { data } = useQuery('test', async () => {
        await sleep(10)
        return 'test'
      })

      return (

{data}

      )
    }

    const rendered = render(
      <ReactQueryCacheProvider queryCache={cache}>
        <Page />
      </ReactQueryCacheProvider>
    )

    await waitFor(() => rendered.getByText('test'))

    expect(cache.getQuery('test')).toBeDefined()
    expect(cache.getQuery('test').config.staleTime).toBe(Infinity)
    cache.clear({ notify: false })
  })

useConfigContext was falling back to defaultConfigRef but when using makeQueryCache, the ref is not updated (on purpose) when merging in defaultConfig.
Now the actual merged configRef for the query cache is exposed so that we can access it in useConfigContext.
Wrong queryCache variable being passed
I fixed another two issues with buildQuery and notifyGlobalListeners that was passing the queryCache module variable instead of this in the new refactored TS code.
Exposing the config ref
~I'm not sure whether you want it exposed off the instance like that or through some other means (since this would also consumers to mess with it?). However, it does address the issue.~
I am exposing it through QueryCache.getDefaultConfig which could be used to freeze or clone the config to prevent mutation.
Exposing it like this would mean:

It's in the public API
It should be added to API docs
People could mess with it

I can document this as a new API, I could freeze it, whatever we think would make sense. We just need to get a reference to the merged queryCache-specific config.
Changes

Use useQueryCache in useConfigContext to fallback to queryCache's configRef
Expose QueryCache.getDefaultConfig() method
Fix passing default queryCache ref in notifyGlobalListeners
Fix passing default queryCache ref in buildQuery

I am currently "integration" testing this in our repo.

Just seeing weird issues locally when testing where the query cache I am creating and providing via ReactQueryCacheProvider isn't actually being used in the component. Not sure yet what is causing that, whether it's a package linking issue or an actual issue with the code.

OK, the issue is that there were two major bugs with makeQueryCache and QueryCache usage because they were unknowingly passing queryCache module variable in a few places accidentally. I will fix those issues in this too.

Alrighty, this is working now as I would expect!

Fantastic!

:tada: This PR is included in version 2.5.10 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


What's the JWT story with Fortune?

Has anyone done any work on this or is there any guidance?
Cheers

I assume you mean JSON Web Tokens? I'm not too familiar on this topic, but why not use this library: https://github.com/auth0/node-jsonwebtoken
Any sort of authorization scheme is business logic, and won't be handled by this library. Here's how you could generically handle authorization checks in Node.js:
import fortune from 'fortune'
import http from 'http'

const store = fortune.create()
const listener = fortune.net.http(store)

http.createServer((request, response) => {
  if (request.headers['authorization'] !== 'foobar') {
    response.statusCode = 401
    return response.end()
  }
  return listener(request, response)
}).listen(1337)

It's just not within the scope of this project to implement authorization.

OK thank you
On Fri, 4 Sep 2015 at 13:29 ████<EMAIL_ADDRESS>wrote:

Closed #125 https://github.com/fortunejs/fortune/issues/125.
—
Reply to this email directly or view it on GitHub
https://github.com/fortunejs/fortune/issues/125#event-401082711.

Fortune is not an oauth server or authorizer.
If you use express/koa, you have to implement your own auth/acl stack, take a look at [oauth2orize] (https://github.com/jaredhanson/oauth2orize).
After that you can wrap fortune.net into a middleware.


Mobile Issues

We are unable to swipe to scroll content on touch devices within the popup overlay. It scrolls fine on desktop browsers and content that exceeds the screen height will display a scrollbar but will not move on mobile. We've tried everything within our build before using one of the demos and finding it also has the same issue.

Closing this one. The issue was reproducible on very old Android devices that don't support position:fixed.


scripts don't work

when i use python scripts, it have no response, and when i load again, it crashed, just like last issues has said.
when i load  java scripts, program tell me that i should use javac to complile it(which script i load is Example1Getting),so i load the Example1Getting.class, and program tell me that i didn't have a java env, so i tell program where the jvm.dll is, then it loading and loading and crashed.
maybe is the problem of java version?

Same issue with me . Unable to run any script . Nothing happen first time script use and using second time crash the gda . Even latest 3.72 also crashing in same way

java and python env must be 32-bit. The best Java env is java4.0-9.0 and the best python env is python2.7-3.6. I will check other bugs about the crash.

Same issue with me . Unable to run any script . Nothing happen

我使用python3.6.8,win7x86,运行GdaMain.py,Nothing happened.out.txt也没用看到


Cannot find config.type.ts

mp3-mediarecorder version: 4.0.3

Error log when building Angular project which uses mp3-mediarecorder:
Error: node_modules/mp3-mediarecorder/worker/index.d.ts:1:33 - error TS2307: Cannot find module '../types/config.type' or its corresponding type declarations.

1 import { Mp3WorkerConfig } from '../types/config.type';

My package.json states that mp3-mediarecorder version 4.0.3 is being used. I see the config.type.ts file in this repo, but it is not present in my local node_modules/mp3-mediarecorder directory.

:tada: This issue has been resolved in version 4.0.5 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

HI there, sorry for the long wait, should be fixed in 4.0.5 ! Cheers


Performing GMRQ

At the stage of  Gmrq.ipynb of your tutorial I have gotten the following errors:
Input In [1], in <cell line: 87>()
84 no_clusters=[700,800,900]
85 tica_lagtime=[2,4,6]
---> 87 GMRQ(no_of_features,no_of_components,no_clusters,tica_lagtime)
Input In [1], in GMRQ(no_of_features, no_of_components, no_clusters, tica_lagtime, parameter, gmrq_dir, clustering_dir)
56 else:
57     print('Running the {}th cycle'.format(i+1))
---> 58 cv = KFold(len(trajectories), n_folds=6, shuffle=True)  #split the dataset into training set and test set
59 results = []
60 print_results = [] #train score, test score
TypeError: init() got an unexpected keyword argument 'n_folds'
Any suggestions?
thanks
Sandor

sklearn.cross_validation  is not working
I have changed it to
sklearn.model_selection

Were you able to solve the issue? I tried with a newly-created conda environment with my script i.e. conda env create -n msmbuilder -f environment.yml and  GMRQ.ipynb works fine on my end.
This tutorial was using a very old scikit learn and things might be quite different in newer version.

I was not able to resolve the problem with changing the n_folders to n_splits
I still have the error:
Input In [1], in <cell line: 87>()
84 no_clusters=[700,800,900]
85 tica_lagtime=[2,4,6]
---> 87 GMRQ(no_of_features,no_of_components,no_clusters,tica_lagtime)
Input In [1], in GMRQ(no_of_features, no_of_components, no_clusters, tica_lagtime, parameter, gmrq_dir, clustering_dir)
56 else:
57 print('Running the {}th cycle'.format(i+1))
---> 58 cv = KFold(len(trajectories), n_splits=6, shuffle=True) #split the dataset into training set and test set
59 results = []
60 print_results = [] #train score, test score
TypeError: init() got multiple values for argument 'n_splits'
--

Dr. Sandor Lovas,
Professor
Associate Editor, Frontiers in Chemical Biology
Department of Biomedical Sciences
Creighton University, School of Medicine
2500 California Plaza
Omaha, NE 68178, USA
Phone:<PHONE_NUMBER>
Fax:<PHONE_NUMBER>
On Mon, 2023-06-05 at 17:13 -0700, Andrew Yik wrote:
Were you able to solve the issue? I tried with a newly-created conda environment with my script i.e. conda env create -n msmbuilder -f environment.yml and GMRQ.ipynb works fine on my end.
This tutorial was using a very old scikit learn and things might be quite different in newer version.
—
Reply to this email directly, view it on GitHubhttps://github.com/ykhdrew/qMSM_tutorial/issues/1#issuecomment-1577716733, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A4CNJD5Z2C4YUN56722Q6QTXJZY4DANCNFSM6AAAAAAY3PR3GY.
You are receiving this because you authored the thread.Message ID: @.***>

KFold no longer takes data at the first position as in this tutorial. The KFold line should read as:
kf = KFold(n_splits=6, shuffle=True)
Then in the for loop:
for fold, (train_index, test_index) in enumerate( kf.split( list( range( len( trajectories ) ) ) ) ):


interface.py

used inbuilt function and simplified some statements

Hi @Praveensenpai,
Went through this. I should document this better, but we're using black as an auto code formatter. This takes any "creative" style out of code styling and eliminates the need for any sort of style related PRs (at least for the source itself. You can try this yourself with (in the root of the repo):
pip install pre-commit
pre-commit run --all-files

The pre-commit configuration is in .pre-commit-config.yaml, and we use that to automate any code styling. If you install pre-commit as a "pre-commit hook" it will run automatically before pushing your code. For example:
$ pre-commit install
$ commit -am "my new feature"
black................................................(no files to check)Skipped
isort................................................(no files to check)Skipped
flake8...............................................(no files to check)Skipped
codespell................................................................Passed
pydocstyle...........................................(no files to check)Skipped

Knowing that, would you mind resubmitting this PR after running black via pre-commit?


Detect system color mode on Firefox

It looks like there is a problem while detecting system color mode on Firefox.
It says that system color mode is light but it isn't.
You can test it on your provided playground, here it is.
OS - Ubuntu 22.04.4
Firefox version - 128.0.3

I don't think prefers-color-scheme is passed to the iframe / app of the stackblitz example in Firefox. It should, however, work when running the example locally (it does for me). If not, maybe your firefox settings is blocking it somehow. Make sure that privacy.resistfingerprinting is set to false.
Note: Firefox does not yet support the sec-ch-prefers-color-scheme header so it will not be able to detect the preferred color scheme on server side. You can see the supported browsers here.


Use joinable! macro to join using non PK columns

Trying to use the joinable! macro for this issue on crates.io - https://github.com/rust-lang/crates.io/pull/740
We're trying to join the following tables -
crate_downloads
(
  crate_id integer NOT NULL,
  downloads integer NOT NULL,
  date date NOT NULL,
  CONSTRAINT crate_downloads_pkey PRIMARY KEY (crate_id, date),
)

CREATE TABLE public.crate_owners
(
  crate_id integer NOT NULL,
  owner_id integer NOT NULL,
  owner_kind integer NOT NULL,
  CONSTRAINT crate_owners_pkey PRIMARY KEY (crate_id, owner_id, owner_kind),
)

Note - removed few columns for clarity.
I want to join these two tables on just the crate_id column. But this is part of a composite key on crate_downloads. So when we do  -
joinable!(crate_owners -> crate_downloads (crate_id));

I get the error -
error[E0271]: type mismatch resolving `<diesel::expression::nullable::Nullable<(schema::crate_downloads::columns::crate_id, schema::crate_downloads::columns::date)> as diesel::Expression>::SqlType == diesel::types::Nullable<diesel::types::Integer>`
   --> src/user/mod.rs:391:5
    |
391 |     joinable!(crate_owners -> crate_downloads (crate_id));
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected tuple, found struct `diesel::types::Integer`
    |
    = note: expected type `diesel::types::Nullable<(diesel::types::Integer, diesel::types::Date)>`
               found type `diesel::types::Nullable<diesel::types::Integer>`
    = note: this error originates in a macro outside of the current crate

Appreciate some guidance here. :)

In theory we could allow this, but it would be more true to the associations API for this to go through the crates table, since that is the primary key both foreign keys are referencing. So this is at least partially a duplicate of the issues for adding through associations, but I'm going to leave this open since this is specific to declaring possible joins.

@sgrif it makes sense to go for a through association for this. But, would it not be nice to let the developers choose the appropriate approach here and diesel giving them all options? :)

This was fixed by 5cd4c0c2


Mobile v2 docs update

Summary
Updates to the mobile "Developer setup" topic for Mobile v2.
Additional changes
New shortcodes, tabs and tab, were created to enable tabbed content sections more conveniently. README.md has been updated with these changes.

Hello @neflyte,
Thanks for your pull request! A Core Committer will review your pull request soon. For code contributions, you can learn more about the review process here.

Thank you for the detailed feedback, @enahum! @larkox - I really like your idea of synchronized tabs; however, this isn't something that the tabs support yet (in either the dev or product docs), and it would introduce a new usage paradigm we haven't used across the docs before that would require some additional UX consideration up front before execution. Definitely something we could consider in the future.

@enahum Is it the next release of react-native that Mattermost supports, or is there a released version that is supported?

@enahum Is it the next release of react-native that Mattermost supports, or is there a released version that is supported?

@neflyte I would not specify the version as we are constantly updating it but not necessarily at the next version, if you want just link the package.json of the project

@enahum If there isn't a particular version that is supported, then I may have to remove all of the content that is a duplicate of what's on the React Native site; we don't want to be the maintainers of that set of instructions.
If, however, there are Mattermost-specific parts to the setup instructions, we definitely want to call attention to those.
Are there Mattermost-specific parts to the setup instructions?

Hmm, these instructions tend to change very little in time, I think we had them in our docs just to make it simpler for the reader by not having to jump between sites. 0/5 how you want to handle it

Per the conversation here, the Windows-specific instructions in this doc are being left as-is. Further updates can be done in a separate ticket after Engineering has performed their testing.

This PR has been automatically labelled "stale" because it hasn't had recent activity.
A core team member will check in on the status of the PR to help with questions.
Thank you for your contribution!

Friendly reminder to review this PR, @enahum @Willyfrog. @neflyte will review the merge conflict.

This PR has been automatically labelled "stale" because it hasn't had recent activity.
A core team member will check in on the status of the PR to help with questions.
Thank you for your contribution!


feat: Adding candidate and relationship btw OTel and kiness delivery stream

Relevant information
We are establishing a relationship between OTel and AWS Kinesis delivery Stream by using the ARN as the matching criterion. Users or customers from the OTel service need to manually send the ARN within the cloud.resource_id span attribute.
On the Cloud Monitoring platform side, we are associating tags with the aws.Arn for AWS Kinesis firehouse delivery streams, ensuring that the collector.name is equal to cloudwatch-metric-streams.
The ARN is formatted as follows for a function: arn:${realm}:kinesis:${region}:${accountId}:stream/${StreamName}
Checklist

[ ] I've read the guidelines and understand the acceptance criteria.
[ ] The value of the attribute marked as identifier will be unique and valid.
[ ] I've confirmed that my entity type wasn't already defined. If it is I'm providing an explanation above.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Updated notes on scalability of various estimators

Closes #109

I'm not sure if this is the best approach. The short version is that

All the estimators in dask-ml will work in parallel on distributed arrays.
The Incremental* wrappers will work in serial over the blocks, so they won't benefit from any parallelism, beyond what scikit-learn implements
The meta-estimators like RandomizedSearchCV inherit whatever parallelism the actual estimator implements. If you search over dask_ml.linear_model.LogisticRegression, things should be fine on a large array. If you search over sklearn.linear_model.LogisticRegression not so much.

We should probably have something like that in index.rst but I don't want to over-complicate things too early.
Adding more scalability examples as suggested by @marcelsan would be great (though we'll want to balance this against the time it takes to do a full doc build).
cc @rhiver

Thank you for the clarification and I think this is a good addition to the docs. Are there plans to implement more estimators in dask-ml in the future?


[Parsing error] To be investigated

The following piece of code does not parse:
File['logrotate.conf'] {
  content => template('logrotate/spec.erb'),
}

The root cause to be investigated...

Reviewed the full syntax (See https://docs.puppetlabs.com/puppet/latest/reference/lang_resources_advanced.html)
The resource type, which can be one of:

A lowercase word with no quotes, like file.
A resource type data type, like File, Resource[File] or Resource['file']. It must have a type but not a title.

Currently not handling the second part.

Given It must have a type but not a title. this will be probably require re-writing the entire resource grammar.


SupportsTwoWayBindingForDateValuesWithFormatString fails on German Windows

Environment:

Windows 10 (English)
German language settings

Repro:

Clone Blazor repo
Call build.cmd
Microsoft.AspNetCore.Blazor.Build.Test.RenderingRazorIntegrationTest.SupportsTwoWayBindingForDateValuesWithFormatString fails with Expected: Sun 2018-03-04, Actual: So. 2018-03-04 (So. stands for German Sonntag = Sunday).

Build works if the test mentioned above is skipped.

Thanks for reporting this! Fixed.


CP-97 Add the global static singleton functions API

This change introduces global static methods to mirror the instance methods. Allows for using the API without instantiating an object.
https://kalamuna.atlassian.net/jira/software/c/projects/CP/boards/140?modal=detail&selectedIssue=CP-97

@jazzsequence Thanks! Resolved the conflicts, and fixed linting. :+1:


systemfields: adds a related model field

Adds a related model field which helps serialize/deserialize a related
object to/from the record dictionary. The core is extract from the
PIDField in Invenio-Records-Resources. (closes #246)

Related to work being done in https://github.com/inveniosoftware/invenio-vocabularies/pull/17

Related to work being done in https://github.com/inveniosoftware/invenio-vocabularies/pull/17

Thanks for the review. Fixed comments, and added tests for session merge.

Thanks for the review. Fixed comments, and added tests for session merge.


Add Gradle module for KMP Compose module

Long term goal of this example project is to become a full stack example using Kotlin, so the server is already being implemented using Ktor and a module implementing the front-end using Compose.

https://github.com/JetBrains/compose-multiplatform-template


Should we use a template language for replacements?

Right now, we have placeholdes where we do straight string replacements (search and replace the string {{version}}).
Should we use a template language, ideally an existing one?

Detect errors on malformed variables.

This would require variable-like things to be escaped

Give the user control over date and version formatting

e.g. having an example Cargo.toml that only lists major+minor rather than the full version

The one downside is that some template languages are coupled to web stuff and we'd be pulling in html and uri-specific features (like filters) which would add unnecessary bloat.  I know at least liquid has support for dropping the language's stdlib.  I know cargo-generate is using it for that very reason.  Granted, I am biased in that I also maintain liquid.

As maintainer of handlebars-rust, I had similar thoughts when designing replacements so I decide to use double-bracket variables, which is compatible with handlebars and many languages .
The only downside is, like you said, I feel a template engine is relatively heavy to this little tool.

Give the user control over date and version formatting
* e.g. having an example Cargo.toml that only lists major+minor rather than the full version

You can do this by offering more replacements. In addition to replacing {{version}}, also replace {{version_major}}, {{version_minor}}, etc. Users who don't need this can just ignore it.

Looking at the runtime template options at https://github.com/rosetta-rs/template-benchmarks-rs
handlebars

{{{ is required for skipping html escaping
Helpers are not optional
A bit verbose for one off rendering

tera

Supports disabling builtins (unsure how much is considered built-in)
Has easy one-off rendering

liquid

Supports disabling all filters, blocks, and tags, leaving pretty much just variable access
Relatively straightforward rendering
Might be able to have a version usable as {{ version }} and {{ version.major }}

As for compile times of each of these, I'm unsure.  Most things at rosetta-rs do compile times but not the template one.

Overall, this has me leaning towards liquid as a slight step up with minimal breaking changes and without committing to too much language-specific features.

Not sure whether to add this here vs. open a new issue, lmk if a new issue is better.
One thing that would be useful with templating is the ability to add custom commands. For example, in our release checklist, we execute this and paste the result:
echo "This release has $(git rev-list --count $(git rev-list --tags --max-count=1)..) commits from $(git shortlog --summary $(git rev-list --tags --max-count=1).. | wc -l | tr -d '[:space:]') contributors. Selected changes:"

...and would be nice to have that done automatically, with something like a {{ ! $cmd }} syntax...


⚠ DesiTorrents has degraded performance

In 0a6f74e, DesiTorrents ($DST) experienced degraded performance:

HTTP code: 200
Response time: 3182 ms

Resolved: DesiTorrents performance has improved in ab02c86.


fix java compilation error

#6399

there has got to be a better way to fix this.

look for CRLF not just CR

done

fixed on this #6257 PR


feat: add support for supplying member_role_id in project members

Adds basic support for custom roles, by allowing Ultimate users to specify role Ids for their Members
Relates to #673

Can't actually assert the change is successful because GitLab don't expose the member_role_id in a get/list call. :(

Known bugs

Adding a member_role_id which is not present results in a 500
When testing we must follow the steps to create a custom role at instance level: https://docs.gitlab.com/ee/user/custom_roles.html#create-a-custom-role

Additional features

REST APIs look like they exist for creating Custom roles on both SaaS and Dedicated: https://gitlab.com/groups/gitlab-org/-/epics/11851 (WIP)

We would need to document clearly how users can create Custom Roles in GL using gitlabform
Currently there is no python-gitlab support for member_roles or group/member_roles

I can fudge the test locally to add user with member_role_id, but creating a custom role on my local instance

Only available in Ultimate, so I have provided a new action in the workflows to support this, and added a secret with an EE Ultimate License I got recently from GitLab using the handbook process.

Not sure how to test this as workflows only get run if they exist on main -> the tests run correctly on my machine with the same Ultimate License I put in a secret used to stand up a local container


Support initializer as inference request input - onnxruntime backend

Background of this PR can be found here: https://github.com/triton-inference-server/server/issues/5489
Essentially, in onnxruntime, initializers (model weights) can be specified as input to override the default weights. The initializers are different in the sense they don't have batch dimension, so changes need to be made in this repo to make sure the shape treatment is correct. In addition, because model weights will be overridden, dynamic batching should be disabled when initializers are to be fed.

Hi @GuanLuo, thanks for the feedback! I have resolved all of them, I'll drop the PR in the core rpo as you suggested. Please let me know if there are any other questions.

If you haven't already, do you mind to follow the contribution page here to submit a signed CLA?

Thanks @GuanLuo !
I have a sample model you can use to add to test: dummy.zip
Model creation code
import onnx
import torch

import numpy as np
import torch.nn as nn

class SimpleNet(nn.Module):

    def __init__(self):
        super(SimpleNet, self).__init__()
        self.fixed = nn.Parameter(torch.arange(4, dtype=torch.float32))
        self.var = nn.Parameter(torch.rand(4, dtype=torch.float32))

    def forward(self, x: torch.Tensor):
        x = x + self.fixed + self.var
        return x

model_name = 'dummy.onnx'

torch.onnx.export(
    SimpleNet(),
    torch.randn(1, 4),
    model_name,
    input_names=['in'],
    output_names=['out'],
    dynamic_axes={'in': {0: 'bs'}, 'out': {0: 'bs'}},
)

model = onnx.load(model_name)

for initializer in model.graph.initializer:
    if initializer.name == 'var':
        model.graph.input.append(onnx.helper.make_tensor_value_info(initializer.name, initializer.data_type, initializer.dims))

onnx.save(model, model_name)

import onnxruntime as ort

sess = ort.InferenceSession('dummy.onnx', providers=['CPUExecutionProvider'])

in_tsr = np.random.randn(2, 4).astype(np.float32)
print(in_tsr)

var_tsr = np.arange(4).astype(np.float32)
print(var_tsr)

print(sess.run(None, {'in': in_tsr, 'var': var_tsr}))

Triton model config
name: "init_ort"
platform: "onnxruntime_onnx"
max_batch_size: 0
default_model_filename: "dummy.onnx"

input [
  {
    name: "in"
    data_type: TYPE_FP32
    dims: [-1, 4]
  },
  {
    name: "var"
    data_type: TYPE_FP32
    dims: [4]
    optional: true
  }
]
output [
  {
    name: "out"
    data_type: TYPE_FP32
    dims: [ -1, 4]
  }
]

optimization {
  graph : {
    level : 1
  }
}

instance_group [
  {
    count: 1
    kind: KIND_GPU
    gpus: [ 0 ]
  }
]

parameters { key: "execution_mode" value: { string_value: "1" } }
parameters { key: "cudnn_conv_algo_search" value: { string_value: "0" } }

Triton test code
import fire

import numpy as np
import tritonclient.http as httpclient

def main():
    triton_client = httpclient.InferenceServerClient(
        url='<IP_ADDRESS>:8000',
        verbose=False,
        insecure=True)

    inputs = []

    in_tsr = np.random.randn(2, 4).astype(np.float32)
    inputs.append(httpclient.InferInput('in', in_tsr.shape, 'FP32'))
    inputs[-1].set_data_from_numpy(in_tsr, binary_data=False)
    print(f'in: {in_tsr}')

    var_tsr = np.arange(4).astype(np.float32)
    inputs.append(httpclient.InferInput('var', var_tsr.shape, 'FP32'))
    inputs[-1].set_data_from_numpy(var_tsr, binary_data=False)
    print(f'var: {var_tsr}')

    outputs = []
    outputs.append(httpclient.InferRequestedOutput('out', binary_data=False))

    try:
        results = triton_client.infer(
            'init_ort',
            inputs,
            outputs=outputs,
        )
        actual_out = results.as_numpy('out')
        print(f'out: {actual_out}')
        expect_out = in_tsr + var_tsr + np.arange(4)
        all_close = np.allclose(actual_out, expect_out)
        print(f'actual_out == expect_out: {all_close}')
    except Exception as e:
        print(e)
        return
    print('Done')

if __name__ == '__main__':
    fire.Fire(main)

Test output
in: [[ 1.7549193  -0.9423713   0.3669717   0.20170133]
 [-0.26022    -0.05359299  0.26156545  0.7463688 ]]
var: [0. 1. 2. 3.]
out: [[ 1.7549193  1.0576286  4.366972   6.201701 ]
 [-0.26022    1.9464071  4.261565   6.746369 ]]
actual_out == expect_out: True
Done

Please let me know if you need anything else. Thanks!

Regarding the CLA, let me double check with my colleagues and circle back with an answer.

Hi @GuanLuo, just had a discussion with our legal colleagues, they have some concerns about the scope of the CLA, I asked them if they think it's ok that the PR is made by an NVIDIA engineer, and they told me it would be preferred that way. Do you think it's OK that you create a PR on this matter and I drop mine? I hope we can merge this PR sooner.

Okay, I will port the changes over after I tested the change, I am working on adding test case based on the model generation script you provided

@taoisu The mirrored PR with minor fixes has been merged, feel free to close this PR. https://github.com/triton-inference-server/onnxruntime_backend/pull/180

This is awesome! Thanks @GuanLuo!


Implement getting binary path on OpenBSD

What does this PR do?
This commit adds a mostly untested function to get the path to the running binary on OpenBSD.
https://stackoverflow.com/questions/31494901/how-to-get-the-executable-path-on-openbsd
Why was it initiated? Any relevant Issues?
I initiated the PR upon request from here: https://github.com/Nuitka/Nuitka/issues/662#issuecomment-2118743049
I'm sorry I can't really help out more, I just did this as a quick hack to get my own thing working so I'm sure it needs to be revised before merging.
PR Checklist

[ ] Correct base branch selected? Should be develop branch.
[ ] Enabled commit hook or executed ./bin/autoformat-nuitka-source.
[ ] All tests still pass. Check the Developer Manual about Running the Tests. There are GitHub
Actions tests that cover the most important things however, and you are welcome to rely on those,
but they might not cover enough.
[ ] Ideally new features or fixed regressions ought to be covered via new tests.
[ ] Ideally new or changed features have documentation updates.

So, I am able to force push into your repository there. I rebased to develop, and commented out with #if 0 what I think is a strip of the directory name that should make it wrong. Please compile with --standalone and report what happens. @SuperDisk

Make sure do git pull --rebase on your branch btw

@kayhayen As far as I know that call you commented out just removes the trailing slash. Is there a sample program I can compile to verify the output against what it's supposed to look like?

tests/standalone/CtypesUsing.py

(building-nuitka-venv) nickfaro$ pwd
/home/npfaro/making-nuitka/Nuitka/CtypesUsing.dist
(building-nuitka-venv) nickfaro$ ./CtypesUsing.bin
(building-nuitka-venv) nickfaro$

It doesn't crash at least, is it supposed to do something?

Oh yes, it is only tested on Windows and Linux really, use tests/standalone/LxmlUsing.py with lxml installed of course then. Not crashing is very good though. :)

Well I mean is there something that actually tests this binary-name-getting functionality?

Standalone does have to use it at program launch I think, cannot work without it, I believe. But you can always just print(__file__) and compile that.

Did you have a chance to try printing file ?

Can you try the current factory branch: You can try it out by going here: https://nuitka.net/doc/factory.html

I'm currently getting this:
(building-nuitka-venv) nickfaro$ more hi.py
print(__file__)
(building-nuitka-venv) nickfaro$ rm hi.bin
(building-nuitka-venv) nickfaro$ python3 -m nuitka hi.py
Nuitka-Options: Used command line options: hi.py
Nuitka-Options:WARNING: You did not specify to follow or include anything but main program. Check options and make sure
Nuitka-Options:WARNING: that is intended.
Nuitka-Options: Detected static libpython to exist, consider '--static-libpython=yes' for better performance, but errors
Nuitka-Options: may happen.
Nuitka: Starting Python compilation with Nuitka '2.3rc5' on Python '3.10' commercial grade 'not installed'.
Nuitka: Completed Python level compilation and optimization.
Nuitka: Generating source code for C backend compiler.
Nuitka: Running data composer tool for optimal constant value handling.
Nuitka: Running C compilation via Scons.
Nuitka-Scons: Backend C compiler: clang (clang 16.0.6).
In file included from static_src/CompiledFunctionType.c:3121:
In file included from /home/npfaro/making-nuitka/building-nuitka-venv/lib/python3.10/site-packages/Nuitka-2.3rc5-py3.10.egg/nuitka/build/static_src/CompiledCodeHelpers.c:56:
/home/npfaro/making-nuitka/building-nuitka-venv/lib/python3.10/site-packages/Nuitka-2.3rc5-py3.10.egg/nuitka/build/static_src/HelpersFilesystemPaths.c:80:17: error: use of undeclared identifier 'ok'
                ok = 1;
                ^
1 error generated.
Nuitka-Scons: Backend linking program with 7 files (no progress information available for this stage).
scons: *** [static_src/CompiledFunctionType.o] Error 1
FATAL: Failed unexpectedly in Scons C backend compilation.
Nuitka:WARNING:     Complex topic! More information can be found at
Nuitka:WARNING: https://nuitka.net/info/scons-backend-failure.html
Nuitka-Reports: Compilation crash report written to file 'nuitka-crash-report.xml'.

Ok, please retry, my bad.

@kayhayen It's working for me :)

This will be in the 2.3 release shortly


Update build-real-world-app.md

Docs Build status updates of commit 0ab0c8c:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

msteams-platform/onboard-teams-dev-platform/build-your-first-app/build-real-world-app.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


SEOImages: User Story 1463880, Part 7

Summary
Updating image file names and references to improve SEO.
Fixes #1463880

@mairaw Please review this PR. There are no warnings and all extensions are lowercase,


Visual Studio startup can be negatively impacted by credential_process based profiles

In https://github.com/aws/aws-toolkit-visual-studio/issues/93#issuecomment-680975507 and https://github.com/aws/aws-toolkit-visual-studio/issues/93#issuecomment-680320588, @cal5barton reports that a shared credentials profile making use of credential_process can cause IDE-wide delays at startup. Presumably whenever the Toolkit refreshes its credentials list as well.
In this situation, the credential process is running something that is prompting the user and waiting for them to type something in.
Currently, the toolkit is unconditionally resolving all credential_process profiles.
Sample config profile entry:
[profile okta-prod-proxy]
credential_process = aws-okta exec okta-prod -- cmd /c C:\batch\proxy-aws-okta-result.bat
region = us-west-2

@cal5barton I can't offer a time frame, but if we were to make some improvements in this space, would you be interested in trying out an alpha build before the change is released?

@cal5barton I can't offer a time frame, but if we were to make some improvements in this space, would you be interested in trying out an alpha build before the change is released?

Yes I can do that.

Two improvements have been made in this space...
Earlier this week we released version <IP_ADDRESS>. This version no longer tries to resolve all of the supported credentials when the Toolkit starts up, improving the startup time.
Today we released version <IP_ADDRESS>. Normally the AWS Explorer will try to reconnect to the credentials profile previously used in your last Visual Studio session, unless the credentials require interaction (like SSO or MFA). This version will no longer automatically reconnect to credential_process based credentials as well, because these could invoke longer running processes, which impact the startup performance. Instead, the AWS Explorer will indicate that the credentials have not been connected, and provides you with a link to establish the connection.


Deprecated?

Hello!
Is seems that this bot you wrote does not work anymore. The mumble library is deprecated and does not work with newer versions of nodeJS. I made another bot here (https://git.lolcat.ca/lolcat/rena_mumble) and it would mean alot to me if you could add a link to it inside the readme?
Many thanks!!

Currently addressing these issues as part of https://github.com/brooksmckinley/mumble-music/issues/21


⚠ Bofrost.de has degraded performance

In 61c89b6, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1127 ms

Resolved: Bofrost.de performance has improved in 7639433 after 28 minutes.


CanMessage issues

I'm having two separate issues with CanMessages. First, when reading in a message and calling the getCanId function, the CanId is off but barely. For example reading in a CanMessage with id = 0x18022080, the returned CanId from getCanId is 0x98022080. Similarly when sending a CanMessage, the data being sent is off as well. Example: The data to be sent should be 60 00 12 04 00 00 00 in hex, but what is being send is 70 17 B0 04 00 FF 00. Any ideas on what could be going on?

Here's my source code, stripped down.
`#include <CanDriver.hpp>
#include <exceptions/CanException.hpp>
#include <exceptions/CanInitException.hpp>
#include <exceptions/InvalidSocketException.hpp>
#include <exceptions/CanCloseException.hpp>
using sockcanpp::CanDriver;
using sockcanpp::CanId;
using sockcanpp::exceptions::CanException;
using sockcanpp::exceptions::CanInitException;
using sockcanpp::exceptions::InvalidSocketException;
using sockcanpp::CanMessage;
enum {
STATE1      = 0x8C602080,
STATE2      = 0x8C612080,
STATE3      = 0x8C622080,
STATEA      = 0x8C702080,
STATEB      = 0x8C712080,
STATEC      = 0x8C722080,
HEARTBEAT   = 0x98022080,
SEND1       = 0x0C648020,
ACK         = 0x0CE88020

};
double          x = 60;
double          y = 120;
unsigned char   z = 1;
bool            a = 0;
bool            b = 0;
}
void sendFrame( CanDriver* &driver ) {
can_frame transmit_message;
transmit_message.can_dlc = 8;

unsigned short tmp = static_cast<unsigned short>( x / 0.01 );
memcpy( &transmit_message.data[0], &tmp, sizeof( tmp ) );
tmp = static_cast<unsigned short>( y / 0.1 );
memcpy( &transmit_message.data[2], &tmp, sizeof( tmp ) );

transmit_message.data[4] = 0;
transmit_message.data[4] = ( static_cast<unsigned char>( z ) << 2 );
transmit_message.data[4] += ( static_cast<unsigned char>( a ) << 1 );
transmit_message.data[4] += ( static_cast<unsigned char>( b ) );

transmit_message.can_id = SYS_CONT_DATA1;

CanMessage sys_cont_data1 ( transmit_message );
driver->sendMessage( sys_cont_data1 );

}
void receiveCanFrame( CanDriver* &driver ) {
if ( !driver->waitForMessages( ) ) {  }

  std::queue<CanMessage> CanMessages = driver->readQueuedMessages( );

  while( !CanMessages.empty( ) ) {

      auto msg = CanMessages.front( );
      CanMessages.pop( );

      if( msg.getCanId( ) == HEARTBEAT ) {

          try {

            CanMessage send_ACK ( 0x0CE88020, "ACK" );
            driver->sendMessage( send_ACK );

            return;

          } catch ( CanException& ex ) {

            std::cerr << "Failed to send ACK Message (void receiveCanFrame( )): " << ex.what( ) << std::endl;

          }

      }

}
int main( ) {
CanDriver* canDriver;
canDriver = new CanDriver( "can0", CAN_RAW );

while ( true ) {

  try {

      sendFrame( canDriver );

      receiveCanFrame( canDriver );

  } catch ( CanInitException& ex ) {

      std::cerr << "An error occurred while initializing CanDriver: " << ex.what( ) << std::endl;

  } catch ( CanException& ex ) {

    std::cerr << "CanException: " << ex.what( ) << std::endl;

  } catch ( InvalidSocketException& ex ) {

    std::cerr << "InvalidSocketException: " << ex.what( ) << std::endl;

  }

}

}`

Here's source code, stripped down.
`enum {
STATE1      = 0x0C602080,
STATE2      = 0x0C612080,
STATE3      = 0x0C622080,
STATEA      = 0x0C702080,
STATEB      = 0x0C712080,
STATEC      = 0x0C722080,
HEARTBEAT   = 0x18022080,
SEND1       = 0x0C648020,
ACK         = 0x0CE88020

};
double          x = 60;
double          y = 120;
unsigned char   z = 1;
bool            a = 0;
bool            b = 0;
}
void sendFrame( CanDriver* &driver ) {
can_frame transmit_message;
transmit_message.can_dlc = 8;

unsigned short tmp = static_cast<unsigned short>( x / 0.01 );
memcpy( &transmit_message.data[0], &tmp, sizeof( tmp ) );
tmp = static_cast<unsigned short>( y / 0.1 );
memcpy( &transmit_message.data[2], &tmp, sizeof( tmp ) );

transmit_message.data[4] = 0;
transmit_message.data[4] = ( static_cast<unsigned char>( z ) << 2 );
transmit_message.data[4] += ( static_cast<unsigned char>( a ) << 1 );
transmit_message.data[4] += ( static_cast<unsigned char>( b ) );

transmit_message.can_id = SYS_CONT_DATA1;

CanMessage sys_cont_data1 ( transmit_message );
driver->sendMessage( sys_cont_data1 );

}
void receiveCanFrame( CanDriver* &driver ) {
if ( !driver->waitForMessages( ) ) {  }

  std::queue<CanMessage> CanMessages = driver->readQueuedMessages( );

  while( !CanMessages.empty( ) ) {

      auto msg = CanMessages.front( );
      CanMessages.pop( );

      if( msg.getCanId( ) == HEARTBEAT ) {

          try {

            CanMessage send_ACK ( 0x0CE88020, "ACK" );
            driver->sendMessage( send_ACK );

            return;

          } catch ( CanException& ex ) {

            std::cerr << "Failed to send ACK Message (void receiveCanFrame( )): " << ex.what( ) << std::endl;

          }

      }

}
int main( ) {
CanDriver* canDriver;
canDriver = new CanDriver( "can0", CAN_RAW );

while ( true ) {

  try {

      sendFrame( canDriver );

      receiveCanFrame( canDriver );

  } catch ( CanInitException& ex ) {

      std::cerr << "An error occurred while initializing CanDriver: " << ex.what( ) << std::endl;

  } catch ( CanException& ex ) {

    std::cerr << "CanException: " << ex.what( ) << std::endl;

  } catch ( InvalidSocketException& ex ) {

    std::cerr << "InvalidSocketException: " << ex.what( ) << std::endl;

  }

}

}`

Source code, stripped down.
`void sendFrame( CanDriver* &driver ) {
can_frame transmit_message;
transmit_message.can_dlc = 8;

unsigned short tmp = static_cast<unsigned short>( x / 0.01 );
memcpy( &transmit_message.data[0], &tmp, sizeof( tmp ) );
tmp = static_cast<unsigned short>( y / 0.1 );
memcpy( &transmit_message.data[2], &tmp, sizeof( tmp ) );

transmit_message.data[4] = 0;
transmit_message.data[4] = ( static_cast<unsigned char>( z ) << 2 );
transmit_message.data[4] += ( static_cast<unsigned char>( a ) << 1 );
transmit_message.data[4] += ( static_cast<unsigned char>( b ) );

transmit_message.can_id = SEND1;

CanMessage send1 ( transmit_message );
driver->sendMessage( send1 );

}`
`void receiveCanFrame( CanDriver* &driver ) {
if ( !driver->waitForMessages( ) ) {  }

  std::queue<CanMessage> CanMessages = driver->readQueuedMessages( );

  while( !CanMessages.empty( ) ) {

      auto msg = CanMessages.front( );
      CanMessages.pop( );

      if( msg.getCanId( ) == HEARTBEAT ) {

          try {

            CanMessage send_ACK ( 0x0CE88020, "ACK" );
            driver->sendMessage( send_ACK );

            return;

          } catch ( CanException& ex ) {

            std::cerr << "Failed to send ACK Message (void receiveCanFrame( )): " << ex.what( ) << std::endl;

          }

      }

}`

Hi there!
Generally speaking, CAN-IDs on Linux don't just contain the arbitration ID, but also contain information about bus errors and the message type.
In your case, it seems the CAN_EFF_FLAG is set, but the library isn't parsing this out and is returning a "faulty" value.
This definitely seems to be a bug in the library.
Concerning the faulty values being sent; I don't think I've seen this behaviour before.
I'll try and reproduce it.

Okay I think I spoke a bit hastily about the data being sent. I think I am running into an issue where I'm shifting the bits on byte 5, and I don't currently think that is a library issue at the moment.
About the CAN IDs, the error I described has been replicated several times on my end, examples:
Known ID                                               returned from getCanID
0x0C602080                                          0x8C602080
0x0C612080                                          0x8C612080
0x18022080                                           0x98022080
From this it appears that the first digit in the hex values is being shifted twice to the left, changing 0 to 8, and 1 to 9.

I'm currently working on a fix branch to remedy the issue.
The issue is with the CanId type.
I'll link this issue when I'm done with the PR. Because I'm both self-employed and working full-time, I don't always have time to sort these things super quickly. I'm on it, though! :)

No worries at all. Thanks for the help.


Problem with decorator

from blacksheep.server.controllers import Controller, get, post
from blacksheep.cookies import Cookie
from blacksheep.messages import Response
from easy_cryptography.hash.hash_funct import compare_hash
from app.configuration import AUTHORIZED
from models import Doctors
from pony.orm import *

class Home(Controller):

    @get("/")
    def index(self):
        return self.view()

class Patients(Controller):

    @get("/patients")
    def patients(self, login: str, password: str):
        if Doctors.exists(login) and (compare_hash(password, Doctors.get_for_update(login = login).password)):
            patients = Patients.select()
            response = self.view(patients=patients)
            response.set_cookie(Cookie(AUTHORIZED,True))
            return response
        else:
            return "{'message':'Неверный логин или пароль'}"

I got this error
Traceback (most recent call last):
File "c:\program files\python38\lib\runpy.py", line 194, in _run_module_as_main
return run_code(code, main_globals, None,
File "c:\program files\python38\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Program Files\Python38\Scripts\uvicorn.exe_main.py", line 7, in
File "c:\program files\python38\lib\site-packages\click\core.py", line 829, in call
return self.main(*args, **kwargs)
File "c:\program files\python38\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "c:\program files\python38\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "c:\program files\python38\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "c:\program files\python38\lib\site-packages\uvicorn\main.py", line 362, in main
run(**kwargs)
File "c:\program files\python38\lib\site-packages\uvicorn\main.py", line 386, in run
server.run()
File "c:\program files\python38\lib\site-packages\uvicorn\server.py", line 49, in run
loop.run_until_complete(self.serve(sockets=sockets))
File "c:\program files\python38\lib\asyncio\base_events.py", line 616, in run_until_complete
return future.result()
File "c:\program files\python38\lib\site-packages\uvicorn\server.py", line 56, in serve
config.load()
File "c:\program files\python38\lib\site-packages\uvicorn\config.py", line 308, in load
self.loaded_app = import_from_string(self.app)
File "c:\program files\python38\lib\site-packages\uvicorn\importer.py", line 20, in import_from_string
module = importlib.import_module(module_str)
File "c:\program files\python38\lib\importlib_init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 671, in _load_unlocked
File "", line 848, in exec_module
File "", line 219, in call_with_frames_removed
File "C:\Users\1\Desktop\Avicena.\server.py", line 14, in
from app.program import configure_application
File "C:\Users\1\Desktop\Avicena.\app\program.py", line 7, in
from . import controllers  # NoQA
File "C:\Users\1\Desktop\Avicena.\app\controllers_init.py", line 1, in
from .home import Home, Patients
File "C:\Users\1\Desktop\Avicena.\app\controllers\home.py", line 9, in
class Home(Controller):
File "C:\Users\1\Desktop\Avicena.\app\controllers\home.py", line 11, in Home
@get("/")
File "c:\program files\python38\lib\site-packages\pony\orm\core.py", line 5587, in get
return make_query(args, frame_depth=cut_traceback_depth+1).get()
File "c:\program files\python38\lib\site-packages\pony\orm\core.py", line 5569, in make_query
tree = string2ast(gen)
File "c:\program files\python38\lib\site-packages\pony\orm\core.py", line 5516, in string2ast
module_node = parse('(%s)' % s)
File "c:\program files\python38\lib\site-packages\pony\thirdparty\compiler\transformer.py", line 64, in parse
return Transformer().parsesuite(buf)
File "c:\program files\python38\lib\site-packages\pony\thirdparty\compiler\transformer.py", line 165, in parsesuite
return self.transform(parser.suite(text))
File "", line 1
(/)
^
SyntaxError: invalid syntax

It was problem with orm


fix: remove setting the callback name for use-places-autocomplete

closes #748
fixes PROD-SWC-WEB-1NK
What changed? Why?
Since we now add the google maps script without PLACES_AUTOCOMPLETE as callback (that's because it isn't defined) we started getting an undefined callback error when we tried to load a second script, to solve that we're just removing manually setting this name in usePlacesAutocomplete and using the default Function.prototype for all use cases. That should stop rendering multiple script tags.
How has it been tested?

[x] Locally
[ ] Vercel Preview Branch
[ ] Unit test
[ ] Functional test

Change management
type=routine
risk=low
impact=sev5

LGTM


Fix SSH keyfile directory only using last file in dir

The loop was not tasking sshlogin on every keyfile_path, it was iterating through every keyfile_path then at the end of that loop it was calling sshlogin. A missed indent.

Thank you!


Need to add style to datepicker emptyLabel

I don't see any style options for the empty Label on DatePicker. It should be added if not there
emptyLabel={"words"}

I don’t think this case requires spical handling inside the core.


dependency on another package

On my Debian 7 installation I had to install an additional package as mentioned in https://github.com/ariya/phantomjs/issues/10904

Oops.... created this in the wrong role :) Very sorry for that.


build(datasets): make Kedro-Datasets 5.1.0 release

Description
Create a release containing ibis.FileDataset.
The TSC vote was missed for ibis.FileDataset; therefore, this PR will also serve as the TSC vote. If ibis.FileDataset fails the vote, I will close this and create a separate PR to remove it.

I will merge this after I get in a fix for https://github.com/kedro-org/kedro-plugins/pull/893#discussion_r1804798521; please hold merge until then!


Check certificate renewal

Ensure that certbot renewal is in place and functioning correctly.

It would be nice to have an automatic renewal of lets'encrypt certificates.
I followed gandi plugin issue and it seems that this plugin https://github.com/obynio/certbot-plugin-gandi has been updated and is working again. May be it is usable for automatic certs updates followed with a restart of solidcommunity.net and solidcommunity.net:8443.

let'encrypt certificates renewal every 3 months :

6 April 2020 0:30

systemd contains /lib/systemd/system/certbot.timer which makes an automatic certbot renewal every 2 months.
The renewal configuration files are located in /etc/letsencrypt/renewal/
# renew_before_expiry = 30 days
version = 0.40.0
archive_dir = /etc/letsencrypt/archive/solidcommunity.net
cert = /etc/letsencrypt/live/solidcommunity.net/cert.pem
privkey = /etc/letsencrypt/live/solidcommunity.net/privkey.pem
chain = /etc/letsencrypt/live/solidcommunity.net/chain.pem
fullchain = /etc/letsencrypt/live/solidcommunity.net/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = *******************
pref_challs = dns-01,
authenticator = certbot-plugin-gandi:dns
certbot_plugin_gandi:dns_credentials = /etc/letsencrypt/gandi.ini
server = https://acme-v02.api.letsencrypt.org/directory

The remaining issue is to restart the servers pm2 restart all after an update of the certs.
An every 3 weeks cron job would do the trick.

For my personal stuff, I have a daily cron that checks whether certificates are due for renewal, e.g.
0 0    * * * user certbot -n renew

And if the day falls into a period where renewals are possible, it will attempt to renew - this is well ahead of it possibly ever expiring.

the use of Ubuntu snap for certbot is taking too much RAM resources. snapd is running on the background and when updating certbot this creates memory swaps. https://matrix.to/#/!VAJLTawStGAyYwzTTY:gitter.im/$O4l4ActnrIWDRQ7zDucSVOMWY_sSb4f85NFJSdJz65E?via=gitter.im&via=matrix.org
This could be avoided using https://askubuntu.com/questions/1045542/how-to-stop-snapd-from-auto-updating


Redesign the color palette

Background
Currently, we have three color themes: light, dark and black. The black theme was added when DoIt was derived from LoveIt due to my dissatisfaction of the dark theme - it's just not dark enough.
This turns out to be not a good idea #654 #1176 and creates a lot of duplication.
Now
The thing we really need is a properly designed color palette that looks nice, has good accessbility, conveys the correct semantics etc. Then it would be relatively easy to integrate that with Tailwind CSS. Hopefully we can make the dark theme better and get rid of the black theme completely.
The problem is that I have no idea how to design a good color palette so I am inclined to just borrow https://primer.style/foundations/primitives/color and make some adjustments on top of it.
Any help or suggestions will be appreciated.

I think borrowing the design from GitHub is good. I love GitHub dark theme and personally use Dark dimmed for Night time

Color CSS variables maybe a good idea.
Like this https://github.com/alex-shpak/hugo-book/blob/master/assets/themes/_auto.scss
This seems to make it easier to expand or delete unnecessary themes. Example https://github.com/rebron1900/11ty-book/tree/master/src/assets/sass/themes

I have made some attempts, but it has to be said that it is more complicated than expected. I change a color $global-background-color  in this commit .

https://github.com/kkbt0/DoIt/commit/8db166613b3adebd8242dbf15e110c645087ecb4
https://github.com/kkbt0/DoIt/commit/69b6e0a85dd65f46e73d56cdee0ef084267c35c2

Sass is .single .content kbd { style } . But html template don't have any kbd . This may be for some input box styles in JavaScript.
So we need https://github.com/tailwindlabs/tailwindcss-typography to add style to any vanilla HTML we don’t control.

And .search-dropdown .dropdown-menu class  depend on CSS Descendant combinator. Class .dropdown-menu  is the style required by autocomplete.min.js .
So we need a custom plugins like this. Ensure that the Tailwindcss style can cover the HTML generated by JavaScript.
https://github.com/tailwindlabs/tailwindcss/discussions/7541
https://play.tailwindcss.com/tWrJX5Rapi
Similarly,  DoIt/assets/lib/valine/Valine.scss .v[data-class=v]  .vcards .vcard .vcontent.expand:after  is also hard to use Tailwindcss  color class.

Thanks for looking into this and raising the challenge. I agree that Tailwind does not work very well with foreign contents, and I don't think we have to move 100% to Tailwind - a mix between Tailwind and SCSS setup may be the best solution here.
I think we can still centralise the color palette using CSS variables, what you've done in https://github.com/kkbt0/DoIt/commit/8db166613b3adebd8242dbf15e110c645087ecb4 is good. Then, for elements that directly present in the HTML that we control, we use Tailwind to style their color, and for foreign content, we can keep our SCSS setup like what you have done here https://github.com/HEIGE-PCloud/DoIt/pull/1253/commits/058c62e5c6bcf7de8bb6ccf331a07d5e68d9df80
I think in this way, we have the best of both worlds.

I think the value of Tailwind mainly reflects in many of the "mostly isolated" components in DoIt. Features like
https://github.com/HEIGE-PCloud/DoIt/blob/a3970ec1af30d027102cd8889701877390d5d8ba/assets/css/_partial/_single/_showcase.scss
https://github.com/HEIGE-PCloud/DoIt/blob/a3970ec1af30d027102cd8889701877390d5d8ba/assets/css/_partial/_single/_sponsor.scss
Can hopefully be decoupled and made future maintenance easier.

Do you like the new design?

It would be great if there is a link where I can check how it looks in the post

https://github.com/HEIGE-PCloud/DoIt/pull/1253
https://do-dun87dsqt-pclouds-projects.vercel.app/
And now it has a smaller css size.


Multiple parallel subscribe actions sometimes crashing app

I had this exception from Bybit.Api library with small improvements to use latest ApiSharp.
Unhandled exception. System.AggregateException: One or more errors occurred. (Collection was modified; enumeration operation may not execute.)
 ---> System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
   at System.Collections.Generic.Queue`1.Enumerator.MoveNext()
   at System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable`1 source, Int32& length)
   at ApiSharp.Models.AsyncResetEvent.<>c__DisplayClass5_0.<WaitAsync>b__0() in E:\Github\ApiSharp\ApiSharp\Models\AsyncResetEvent.cs:line 51
   at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
   --- End of inner exception stack trace ---
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
[1]    174402 IOT instruction (core dumped)

Could be using  simple _lockObject = new object(); for synchronization in AsyncResetEvent?

I think it has been fixed in ApiSharp 2.0.0. Please try ByBit.Api 1.2.0. If you have trouble again please let me know

Hi! 2.0.3. Same issue. Crushing Bybit and Okx.
I think there could be different lock object

inspecting deeply

Yep, sure. There is https://github.com/dnmsk/ApiSharpApiExample
Launch and wait it for some time

I've no idea what happening in that code, but think that follow implementation won't throw concurrent dependent exception
namespace ApiSharp.Models;

/// <summary>
/// Async auto reset based on Stephen Toub`s implementation
/// https://devblogs.microsoft.com/pfxteam/building-async-coordination-primitives-part-2-asyncautoresetevent/
/// </summary>
public class AsyncResetEvent : IDisposable
{
    private static readonly Task<bool> _completed = Task.FromResult(true);
    private readonly List<TaskCompletionSource<bool>> _waits = new();
    //private Queue<TaskCompletionSource<bool>> _waits = new();
    private bool _signaled;
    private readonly bool _reset;

    /// <summary>
    /// New AsyncResetEvent
    /// </summary>
    /// <param name="initialState"></param>
    /// <param name="reset"></param>
    public AsyncResetEvent(bool initialState = false, bool reset = true)
    {
        _signaled = initialState;
        _reset = reset;
    }

    /// <summary>
    /// Wait for the AsyncResetEvent to be set
    /// </summary>
    /// <param name="timeout"></param>
    /// <returns></returns>
    public async Task<bool> WaitAsync(TimeSpan? timeout = null)
    {
        lock (_waits)
        {
            if (_signaled)
            {
                if (_reset)
                    _signaled = false;
                return _completed.Result;
            }
            else
            {
                var tcs = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
                if (timeout != null)
                {
                    var cancellationSource = new CancellationTokenSource(timeout.Value);
                    var registration = cancellationSource.Token.Register(() =>
                    {
                        tcs.TrySetResult(false);

                        // Not the cleanest but it works
                        lock (_waits)
                        {
                            var ourWait = _waits.FirstOrDefault(w => w == tcs);
                            if (ourWait != null)
                            {
                                _waits.Remove(ourWait);
                            }
                        }
                    }, useSynchronizationContext: false);
                }

                _waits.Add(tcs);
                return tcs.Task.Result;
            }
        }
    }

    /// <summary>
    /// Signal a waiter
    /// </summary>
    public void Set()
    {
        lock (_waits)
        {
            if (!_reset)
            {
                // Act as ManualResetEvent. Once set keep it signaled and signal everyone who is waiting
                _signaled = true;
                for (var i = 0; i < _waits.Count; i++)
                {
                    _waits[i].TrySetResult(true);
                }
                _waits.Clear();
            }
            else
            {
                // Act as AutoResetEvent. When set signal 1 waiter
                if (_waits.Count > 0)
                {
                    for (var i = 0; i < _waits.Count; i++)
                    {
                        _waits[i].TrySetResult(true);
                    }
                    _waits.Clear();
                }
                else if (!_signaled)
                    _signaled = true;
            }
        }
    }

    /// <summary>
    /// Dispose
    /// </summary>
    public void Dispose()
    {
        _waits.Clear();
    }
}

I'm in hospital and cant write code or do any test. I will ve able to check it on monday. In my mind

Queue or List is not thread safe structures. I've fixed it with ConcurrentQueue. I tested with your codes again and got no error this time. I'll publish it tonight. Thanks for your feedback and patience

Fixed in Version 2.0.5


Config Implementation + Discovery HUD Config

Adds a config implemented using cloth config.
Cloth config has been JIJ'd for Fabric/Quilt. Forge only loads the configuration if the mod is manually installed.
Configs

openWithZoom - Whether or not the Discovery HUD opens with a zoom animation, matching the spyglass opening animation.
eyeOpens - Whether or not the eye will display as open at any time. Accessibility option for possible triggers.

I decided against simply not rendering the eye as it leaves an empty space in the HUD. This is the simplest way to account for that, and still looks good.

Before I review this, I still think there should be some consistency in the dependencies. I think we should either shadow it in all modloader versions or have it as a dependency in all modloaders. Having it as a dependency for some and built-in to others is needlessly confusing and is going to cause issues for users.


[RFC] Read IP from request attribute

Detecting client IP is a tricky business, so the need for a middleware like https://github.com/middlewares/client-ip/
What do you think about adding an option to read IP parameter from a request attribute?

Good idea.
In fact, there's the same feature in middlewares/firewall that we can replicate.

To replicate it, AccessLog::commonFormat cannot be static. I'll remove static keyword from all methods. What do you think?

Yes, no problem. 👍


Tokenizer refactor

although they work alright (hopefully lol) the tokenizers are still somewhat rudimentary. all of them are brute force solutions to do stuff (greedy encoding during code point level BPE for example). also maybe clean up the codebase a little, especially for fallback-to-unknown-char cases

did refactorization but only code organization, no complexity improvement or algorithm whatsoever


uncaught exception: Highcharts error #13: www.highcharts.com/errors/13

uncaught exception: Highcharts error #13: www.highcharts.com/errors/13
something wrong with js

I am using latest version but its not fixed always showing error #13


Add { descending: true } option

This makes it clearer what true or false means when sorting ascending/descending. It's an API breaking change but a worthy one IMO. I haven't thought too hard about the implementation so if I've missed anything obvious, please let me know and I'll fix it.
Closes #2

@jamwaffles Thank you
I like this idea because it allows us to introduce compareFunction as a parameter which is similar to Array.prototype.sort([optional compareFunction])

Changed! Also added a couple of backwards compatibility tests.


chore: add back cli

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#288  👈
#286
#285
#284
#283
#282
#281
#280
#279
#278
#277
#276
#275
#274
#273
#266
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @NathanFlurry and the rest of your teammates on  Graphite


Propose adding a delimeter to Column()

Right now Columns takes an int, and uses strings.Fields to split the line.  I'd like to propose we add a delimiter, so that you could parse csv, semicolon, or pipe delimited files.  The function would look like this:
func (p *Pipe) Column(col int, d ...string) *Pipe {
	var delim string
	if len(d) > 0 {
		delim = d[0]
	}
	return p.FilterScan(func(line string, w io.Writer) {
		var columns []string
		if delim == "" {
			columns = strings.Fields(line)
		} else {
			columns = strings.Split(line, delim)
		}
		if col > 0 && col <= len(columns) {
			fmt.Fprintln(w, columns[col-1])
		}
	})
}

d is a ...string means it takes 0 or more arguments, so if the user doesn't pass anything it still works maintaining compatibility with existing programs
Ex:
script.Exec(`echo "one two three"`).Columns(2).Stdout()

would still print 2.   However
script.Exec(`echo "one,two,three"`).Columns(2, ",").Stdout()

Would also print 2
Sort of a simple change, and it works for a case where I want to parse through comma separated lines. While I think it works, I might not be thinking of a use case where it would break existing code.

Hey @rmasci Can I work on this issue?

What about something like:
script.File("data.csv").Fields(",").Columns(1, 3, 9).Stdout()

sounds nice to me if we can extend it to CSVs along with strings

I did this. You can specify the delimiter, the fields like 1,3,7, and the
delimiter out. I even added a CSV formatter to convert the CSV to a table
like MySQL, grid, or an HTML table. I wanted to ask if it was wanted before
issuing a pull request. I keep it in my version of 'script' in my github.
On Sat, Sep 7, 2024 at 2:59 PM Rupesh Prajapati @.***>
wrote:

sounds nice to me if we can extend it to CSVs along with strings
—
Reply to this email directly, view it on GitHub
https://github.com/bitfield/script/issues/180#issuecomment-2336406467,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGRIPKUESWPS5O7PHXBXSDZVNEHRAVCNFSM6AAAAABNXBGB4SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZWGQYDMNBWG4
.
You are receiving this because you were mentioned.Message ID:
@.***>

For example:
sample.csv:
Name,Abbreviation,Numeric,Numeric2
Monday,Mon,1,01
Tuesday,Tue,2,02
Wednesday,Wed,3,03
Thursday,Thu,4,04
Friday,Fri,5,05
Saturday,Sat,6,06
Sunday,Sun,7,07
Code:
script.File("sample.csv").Fields(",", "|", 1, 3).Stdout()

Output:
Name|Numeric
Monday|1
Tuesday|2
Wednesday|3
Thursday|4
Friday|5
Saturday|6
Sunday|7
func main() {
script.File("sample.csv").Fields(",", "|", 1, 3).Stdout()
}
On Sat, Sep 7, 2024 at 3:25 PM Richard Masci @.***> wrote:

I did this. You can specify the delimiter, the fields like 1,3,7, and the
delimiter out. I even added a CSV formatter to convert the CSV to a table
like MySQL, grid, or an HTML table. I wanted to ask if it was wanted before
issuing a pull request. I keep it in my version of 'script' in my github.
On Sat, Sep 7, 2024 at 2:59 PM Rupesh Prajapati @.***>
wrote:

sounds nice to me if we can extend it to CSVs along with strings
—
Reply to this email directly, view it on GitHub
https://github.com/bitfield/script/issues/180#issuecomment-2336406467,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGRIPKUESWPS5O7PHXBXSDZVNEHRAVCNFSM6AAAAABNXBGB4SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZWGQYDMNBWG4
.
You are receiving this because you were mentioned.Message ID:
@.***>

Very cool! I'm inspired by nushell to think we should have more data-analysis capabilities like this in script. I don't have any concrete suggestions to make at the moment though. Some use cases would be welcome.

Just my two thoughts: How about making the delimiter variant a separate Method, like .ColumnsDelim()?  And while having a variant taking a rune or string for the second argument is probably nice to have as an easy option, I'd like a .ColumnsRegexp(), too. 🙂

What about this suggested API, to separate the two concerns of parsing the fields and selecting the fields?

Sounds good, I guess this could/should then have variants for different types of delimiter definition mechanisms (string, regexp, maybe a func)?  Speaking of which, having a func option would allow for the regexps I was wishing for to be handled inside that, too, alleviating the need to have a regexp variant.

I did this in my version of script.  I asked at the time if you wanted a pull request but didn't hear anything. This is my version on go.doc:
<EMAIL_ADDRESS>script.File("sample.csv").Fields(",", ",", 2, 4).Stdout() //.Table()

[08:23 PM rmasci@Richs-Mac-mini ~/go/scriptcsv] $ go run main.go
Abbreviation,Numeric-2
Mon,01
Tue,02
Wed,03
Thu,04
Fri,05
Sat,06
Sun,07

If you replacer .Stdout with  .Table() function it formats it for you.
<EMAIL_ADDRESS>$ go run main.go
+-----------------+--------------+
|   Abbreviation  |   Numeric-2  |
+=================+==============+
|       Mon       |      01      |
|       Tue       |      02      |
|       Wed       |      03      |
|       Thu       |      04      |
|       Fri       |      05      |
|       Sat       |      06      |
|       Sun       |      07      |
+-----------------+--------------+


Missing a data persistence state graph

Page
Include the link to the page you are reporting a defect
Summary
There's no public document page illustrating the persistence state shift flow. Having one such graph will help users understand the persistence states. This also helps users trouble-shoot scenarios like LOST files in ASYNCH_THROUGH.

+1 for a developer document detailing valid inode states and state transitions. I've been hit with weird messages such as IllegalStateException cannot transition from TO_BE_PERSISTED to NOT_PERSISTED etc.

Will close it for now, feel free to reopen it and contact us if this is a strong request.


pm: state: improve DT macros

This PR improves the naming, organization and docs for the PM state DT macros. Refer to individual commits for more details.

@ceolin please take a look


python modules in the repository

I've started writing an obspy2mspass python function that will take an obspy Trace object and construct a TimeSeries object from it.   The issue I wanted to ask is two-fold:

Should we start a different "module" for this kind of thing or integrate the code with mspasspy?  I'm not sure if the later is actually possible as it would mix binary .so (dylib) with python scripts.   Might be possible with cython, but that might get even uglier.
How should the repository be laid out to handle a library of python procedures?

That is the other thing I was thinking about today. I was trying to find a good example on how to make CMake work with any build systems of Python. Not sure that there is anyone actually doing such a thing. That's why in https://github.com/wangyinz/mspass/issues/17#issuecomment-562760031 I prefer not to have another python script to handle the template. I think for now, we should always put C++ code in /cxx and Python code in /python.
I am still in the phase of learning the best practice of organizing a python project, so I don't have a good answer to that yet. I think for now, it is OK to just put it under /python. I will reorganize it once I come up with a good plan. I guess it will be fine at this early stage of development.

Keep in mind I'm at about the 200 level in python programming - I know most of the features of the language but am no totally fluent.   That is a caveat that this may be off base.
Was reading about "modules" and "packages" in this online source.   Hadn't quite understood the distinction between the two before.   Made me realize we might want to organize some of this stuff into a package with mspass as the top level name.  That source says this is easy for pure python code; you just put the code for each module in subdirectories.   obpsy does this, which is why they have obspy.core, obspy.db, etc.   Seems to me we should do that too to keep the namespace cleaner.  I'm a bit unsure how to do that with the C code, but doubt it is that hard.   Just need to know he tricks
Here is suggested package.module starting organization:
mspass - let this become the package name.   Alternatively, we could call it mspasspy to make it clear that is the python package.  mspasspy is probably better.
mspasspy.Cseis or just mspasspy.seismic or maybe mspasspy.core ala obspy
This would contain all seismic related code produced from wrapped C++.   One model is the current mspasspy module.  We might consider splitting that module into a seismic and utility (or some similar name) to follow the C source tree organization.  There is logic in that as the utility components have use beyond just seismology.
mspasspy.parallel
Python scripts suitable for map-reduce.  The example.py script you have is a starting point.
mspasspy.processing
Might want a separate module for processing algorithms not linked to spark.   I'm thinking these would be normal function or classes that could process one object at a time.  The idea is map-reduce scripts would typically have calls to these functions/class-methods.
The other key advantage this would provide is a cleaner method for add ons by others as the system develops.  We could use the antelope model and have a mspasspy.contrib or create other generic names for common elements.  e.g. I could see evolution to mspasspy.graphics, mspasspy.simulation, or any number of others.

This needs to be the organization for the python directory. I prefer to still call it mspasspy, since mspass should refer to the whole thing including MongoDB and Spark. The C++ code need to have a different name, since it may not be seismic code only. Maybe we should call it mspasspy.cmodule or just mspasspy.cm.  Then, we can just follow the C source tree organization, and have submodules mspasspy.cm.seismic and mspasspy.cm.utility. Not sure what we wanna do with the mspasspy.cm.algorithm module, since that overlaps with  mspasspy.processing in functionality. Maybe we should name the same which will be mspasspy.cm.processing and mspasspy.processing.

I have uploaded current version of the mspasspy package to PyPI as a test. There is a couple issue that worth noting here. I ran following commands to create the distribution and upload it:
python3 setup.py sdist bdist_wheel
mv dist/mspasspy-0.0.1-cp36-cp36m-linux_x86_64.whl  dist/mspasspy-0.0.1-cp36-cp36m-manylinux2014_x86_64.whl
python3 -m twine upload dist/*

Note that if doing this within the Docker container, we will need to install the following packages:
pip3 install --upgrade twine keyrings.alt

The issue is that I have to change the name of the generated distribution due to PyPI's support of compiled package (refer to PEP 599). So, what I have above it totally a hack. Preferably, we will want to setup a Travis workflow to do the distribution with the Docker image provided here. There are more setups needed though as we have a lot of dependencies to be resolved.

I now have a specific problem that requires a solution to this issue.   We have the base mspasspy module produced from the C++ code and wrappers.
We very much need to start the process of construction of basic functions to load static data (what antelope calls dbmaster) and waveforms.   In my opinion these need to be split into a different module that splits these functions into a functional block.   This particular one should probably be called mspass.import.
How do we build this structure?   A complication is these will likely require a mix of C++ and python code.

Oh, I thought we are only developing the reader and writer in Python. Since we have the C++ wrapped in Python already, can't we just do the import with the MongoDB Python API? What is the need to C++ code here? We could build a similar import module in our C++ API using MongoDB C++ API, but that should be a separate development, right?

If we are following the directory structure that we decided earlier, the mspasspy.import module should be placed in /python/import/.

Ok, that is a clean structure for a python module and agrees with the web source at the head of this issue discussion.
I think I confused the issue with the sentence about mixing C++ and python, but it really comes to a head here.   The issue is that the easiest path to solution for me to write the parser for the yaml representation of the Metadata is to write a C++ program using that package you added earlier (don't remember the name immediately, but the C++ code that parses yaml).  This has a scalability issue I haven't quite figure out how to deal with yet.   The way I wrote the antelope intermediate file is that the yaml file has one block in a list for each seismogram in a data file.   The example I just ran had 1913 seismograms from the first year of USArray data limited to segments defined by P picks made at ANF.   That is a decent starting point, I think.   However, later the number of seismogram could be millions if I put everything in year files and didn't restrict the conversion to the largest events.   100000, at least, would be routine.   I can eat the header data, but not all the waveform data at once.
The problem this raises is that writing the yaml parser should be easy.  I think I can do it initially the lazy way and just have the parser build an std::vectormspass::Metadata container or probably better to use a list as it wouldn't require the data to be in a single memory block.   If you look in the cxx/python directory you will see a function I wrote in C++ that can be called from Python to convert Metadata to a python dict that can be used for CRUD operations in Mongo.  This converter crosses the C++ to python boundary at a nasty point.   Reading the yaml and binary files the antelope converter produced is easiest to do in C++, but the database transactions are best done in python.
I think I only have two options:

parse the yaml file in python - presumably there are good libraries out there - and create a python converter to read the sample data and save the result in MongoDB.
crack the C++ database interface and have this converter be pure C++ with the only objective being to load antelope data.

I suspect #1 is the easier path to solution.  Any suggestions?

PyYAML seems the likely library for a pure python solution.   Any experience with this?

Yes, I can confirm that is the common choice. Should be able to just do a pip install pyyaml and then use it as import yaml.

I agree it is easier to do it in Python with its own yaml parser. This will make it easier to run in parallel with Spark.

I could use some suggestions on how to organize a set of stuff I've been building for what I think could be characterized as preprocessing.   Some functionalities the completed module could and should have are:

Load station xml data downloaded by web services to build site and channel collections (done and solid)
Index miniseed data.  This is a necessary preprocessing step to take raw data assembled from an array of sources before doing what would best be called an import into mspass.   There are separate threads on this topic here #74
Various forms of housekeeping functions to map raw data to a form usable by mspass will be needed.   Much of the current focus is a particular example, but there are likely other related tools that will need to developed that fit in this category. The current problem to build our full up usarray test data set is the obnoxious problem of mapping seed net:sta channel codes to data tagged with only a sta field in antelope processing.   The particular example of the day is css3.0 "arrival" table data that need to be linked to waveform data tagged with net:set.   If we don't do that right we will link the wrong waveforms to the wrong arrival times.   Details aren't the point for this  issues page. The point is there will be a growing class of (often custom) tools for cleaning up metadata problems.   What should we call these and where should they go in the python tree?
I'm ultimately aimed at writing an import function, BUT it probably should be fully integrated into mspass and (a) implement history and (b) be runable as a map function in spark.  It may thus need a different home in the python tree.
There are a set of functions we need to develop for managing source metadata.   We have a few things already that are currently part of the Database class.  I'm not sure that class is correctly designed.
Others?

I think a key point is we need to think about this now before the code base gets too ponderous to change things easily.   I think two key questions are:

What functionality should be part of the Database class and what functionality should be put into a suite of functions?
What structure should the modules for this type of thing have?  Right now we don't have an clear design for python module organization.   As you noted last week ccore may need some fragmentation too.  Can we mix pybind11 generated modules with python modules in the same directory?   I think all that is needed is to give the C++ generated library and the python code files different names right?  (e.g. if we had a dbcpp module under a db directory and had a "preprocessing.py" module in the same directory one could use constructs (I think) like:  import mspasspy.db.dbcpp and import mspasspy.db.preprocessing right?)

Anyway, let's have a discussion here as appropriate for the record in combination with offline verbal discussions in our weekly calls.

Various forms of housekeeping functions to map raw data to a form usable by mspass will be needed. What should we call these and where should they go in the python tree?
I'm ultimately aimed at writing an import function, BUT it probably should be fully integrated into mspass and (a) implement history and (b) be runable as a map function in spark.  It may thus need a different home in the python tree.
There are a set of functions we need to develop for managing source metadata.   We have a few things already that are currently part of the Database class.  I'm not sure that class is correctly designed.
What functionality should be part of the Database class and what functionality should be put into a suite of functions?

These questions are pretty much asking how we should arrange the database module. I think we should move the discussion of API design to our google doc (I already created). We definitely should turn it into offline discussions.

What structure should the modules for this type of thing have?  Right now we don't have an clear design for python module organization.   As you noted last week ccore may need some fragmentation too.  Can we mix pybind11 generated modules with python modules in the same directory?   I think all that is needed is to give the C++ generated library and the python code files different names right?  (e.g. if we had a dbcpp module under a db directory and had a "preprocessing.py" module in the same directory one could use constructs (I think) like:  import mspasspy.db.dbcpp and import mspasspy.db.preprocessing right?)

I think that will work, but we probably want to limit the mix of C++ and Python modules. It will be confusing to the users since the C++ bindings will have some subtle difference that will be unexpected to the users.

Just a note here. After re-structure the pybind11 code, it is pretty straightforward to mix Python and C++ code within a module - just put them under the same directory.