Allowing Configure method to be inherited by Startup class

I'm going to have multiple web applications sharing base code and it would be great if the Startup class could inherit the Configure method from a parent class. Otherwise, I'm going to have to copy and paste the same boilerplate.
e.g. this wouldn't fail
    public class Startup : StartupTemplate
    {
        public override async Task ProcessRequestAsync(HttpContext Context)
        {
            await Context.Response.WriteAsync("Hello World");
        }
    }
    public abstract class StartupTemplate
    {
        public void Configure(IApplicationBuilder app)
        {
            app.Run(ProcessRequestAsync);
        }
        public abstract Task ProcessRequestAsync(HttpContext Context);
    }

This returns the basic Configure not found error:
    System.Exception: TODO: ConfigureDevelopment or Configure method not found
    at Microsoft.AspNet.Hosting.Startup.StartupLoader.LoadStartup(String applicationName, String environmentName, IList`1 diagnosticMessages)

It looks like the update would need to be here: https://github.com/aspnet/Hosting/blob/dev/src/Microsoft.AspNet.Hosting/Startup/StartupLoader.cs
    var methodInfo = startupType.GetTypeInfo().GetDeclaredMethod(methodNameWithEnv)
    ?? startupType.GetTypeInfo().GetDeclaredMethod(methodNameWithNoEnv);

Could GetDeclaredMethod be replaced with GetMethod(String, BindingFlags)?

I ran into this same issue and also believe that not requiring declared methods is a good idea so that we can derive the Startup.cs file from a base class.
Changing the line method above to the following would be nice:
var methodInfo = startupType.GetTypeInfo().GetMethod(methodNameWithEnv, BindingFlags.Public | BindingFlags.Instance)
        ?? startupType.GetTypeInfo().GetMethod(methodNameWithNoEnv, BindingFlags.Public | BindingFlags.Instance);

We are ok with taking this change if either of you guys want to submit a pull request, see the instructions if you haven't done it before.

My first time doing a pull request, so let me know if I did it wrong:  https://github.com/aspnet/Hosting/pull/153

https://github.com/aspnet/Hosting/pull/153 fixed this.


MSPSDS-794: Changes to activity option labels and order

Pull Request Test Coverage Report for Build 2012

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 96.721%

Totals

Change from base Build 2010:
0.0%

Covered Lines:
3953

Relevant Lines:
4087

💛  - Coveralls


brew services and root crash since last brew upgrade

Related to https://github.com/Homebrew/brew/issues/14462


md/vs

[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ed9dcc5d-ade9-c88a-76f6-538231671b0b
Version Independent ID: 6d86c444-dba6-8d0e-4b2c-a4cc57c11365
Content: First look at the debugger - Visual Studio (Windows)
Content Source: docs/debugger/debugger-feature-tour.md
Product: visual-studio-windows
Technology: vs-ide-debug
GitHub Login: @Mikejo5000
Microsoft Alias: mikejo

@Munyola -- Daniel, thank you for your feedback.  Please clarify your intended task or issue.  Meanwhile, consider these resources:

Visual Studio on Q&A
Visual Studio MSDN Forum
Stack Overflow
Reddit


[Connection] remove conn-type udp

Remove enum - UDP connection, nnstreamer-edge handles TCP connection only.

:memo: TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #152. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/.


Create test script for Discord API and Bots

Create a test script for Discord API and Bots scripts

@amitnudel
Adding you to the ticket.
feel free to kill mario

@amitnudel @mario99logic  moving you to the main test ticket.


Removes previous notifications when switching input source

Notification accumulates when switching input sources, you get tons of notifications in the Notification Center at the end of the day, it's quite annoying really.

Looks nice! Thanks.

I'll release this into v1.1.0.

Updated in homebrew cask
https://github.com/caskroom/homebrew-cask/pull/40598

Thanks @TPeterW 👍


convert some service classes to static fields, run [github]

Warnings

:warning:
This PR modified service code for github but not its test code. That's okay so long as it's refactoring existing code.

Messages

:book:
:sparkles: Thanks for your contribution to Shields, @calebcartwright!

  Generated by :no_entry_sign: dangerJS against 98d5a57fbb71929d1c52993dc0fd21d13fe7e15c


React component with windups not working in Astro.

I'm trying to use windups with Astro in a component made with React but, I get this error from Astro.

Any ideas?

No idea. It looks like the import is not working, as the windups module is coming back undefined. Were you able to solve tihs?

I ended up using a different package to achieve my goal, wasn't able to figure out what was breaking my code.


Error: Cannot find module '/home/xxxx/party-js/lib' in Next JS

Describe the bug
When i add party JS to my project, in first time this works ok, but if i reload the page, this the error appear for me

To Reproduce

Add party-js lib
import to your project
implements the party js
reload the page

My code:

Expected behavior

Environment

Additional context

Next JS -> 12.1.6
React and react-dom -> 18.2.0
react-redux -> 8.0.2
redux-persist -> 6.0.0

Can you give some additional context? What version are you on, which framework are you using?

I can reproduce it, but I can't figure out what's causing it, since it's working fine for non-Next.js apps.
I'll look into it more once I have more time on my hands.

Okay! Thanks.. I'm waiting for you

Nice! I will try here

@yiliansource problem solved!


[ROCKETMQ-286] Add a switch to decide whether to wake up commitLogService when put message

Wake up commitLogService will make messages consumed quicker when put message, but it may increase disk usage. So, add a switch to decide where to open this feature, described in https://issues.apache.org/jira/browse/ROCKETMQ-286.
Brief change log
Add a config property to MessageStoreConfig
Verifying this change
Just a switch, verifying is to change it's value.
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a JIRA issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ROCKETMQ-XXX] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exists. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

@zhouxinyu  @lizhanhui  @vongosling   pls have a review.

Coverage decreased (-0.2%) to 38.636% when pulling 6bc0b58c04339c5f24eb306c60b25c6dc8ec626d on vsair:ROCKETMQ-286 into 254d432496c424717d45b8dc9e44ae3bd78ab466 on apache:develop.

@vsair  Could you follow our new PR style for this PR, like https://github.com/apache/incubator-rocketmq/pull/152

@vongosling  I'v modified the description.

@zhouxinyu  @vongosling  pls have a review ?

LGTM, wake up commitLogService is for real-time, but it may let broker suffer from SSD write amplification, so provide a switch is reasonable.

@vsair @zhouxinyu another config flushCommitLogTimed has already handled such situation, and avoided SSD write amplification.
If flushCommitLogTimed is enabled, it will only wakeup the commitLogService, but could not wakeup the flushCommitLogService, for:
 if (flushCommitLogTimed) {
      Thread.sleep(interval);
 } else {
      this.waitForRunning(interval);
}

This PR is for ConsumeQueue, and flushCommitLogTimed is only used by FlushCommitLogService

@vsair What happened in progress

@dongeforever  yeah, it's right, I closed the PR.


fixed reference before assignment bug with EMAIL variable

After checking out your pr, it seems like you forgot to add the import for "re"


feat: backport blocksync metrics

Hi @czarcas7ic -- you'd asked how to get block size, and while I was working, I noticed that is something provided by metrics for blocksync in main.
Also, please feel free to let me know if it is better to make this PR, without all of the other changes in it, that wouldn't be hard.

Add CMT_HOME (or remove it?) (backport #983) (#999)
build(deps): Bump bufbuild/buf-setup-action from 1.21.0 to 1.22.0 (#1029)
docs: Added double quotes to /abci_query path param (#1015) (#1046)
Clarifies that processProposal may be called for set of transactions different from the one returned in the preceding prepareProposal (backport #1033) (#1053)
v0.37.x: backport of new content on spec/p2p (#1004)
build(deps): Bump bufbuild/buf-setup-action from 1.22.0 to 1.23.1 (#1063)
build(deps): Bump docker/setup-buildx-action from 2.7.0 to 2.8.0 (#1062)
build(deps): Bump docker/setup-buildx-action from 2.8.0 to 2.9.0 (#1104)
build(deps): Bump docker/setup-buildx-action from 2.9.0 to 2.9.1 (#1129)
build(deps): Bump bufbuild/buf-setup-action from 1.23.1 to 1.24.0 (#1130)
p2p: Remove UPnP functionality (backport #1114) (#1126)
crypto/sr25519: upgrade to<EMAIL_ADDRESS>(#1151)
build(deps): Bump bufbuild/buf-setup-action from 1.24.0 to 1.25.0 (#1158)
build(deps): Bump bufbuild/buf-setup-action from 1.25.0 to 1.25.1 (#1219)
Close evidence.db OnStop (#1210) (#1225)
Log proposer's address when correctly accepting a proposal (backport #1079) (#1221)
build(deps): Bump bufbuild/buf-setup-action from 1.25.1 to 1.26.0 (#1237)
build(deps): Bump bufbuild/buf-setup-action from 1.26.0 to 1.26.1 (#1274)
build(deps): Bump docker/setup-buildx-action from 2.9.1 to 2.10.0 (#1301)
feat: make handshake cancelable (backport #857) (#1013)
build(deps): Bump docker/build-push-action from 4.1.1 to 4.2.1 (#1314)
build(deps): Bump actions/checkout from 3 to 4 (#1315)
Fix typo: exent -> event (#1329)
build(deps): Bump docker/build-push-action from 4.2.1 to 5.0.0 (#1356)
build(deps): Bump docker/login-action from 2.2.0 to 3.0.0 (#1359)
build(deps): Bump docker/setup-buildx-action from 2.10.0 to 3.0.0 (#1358)
build(deps): Bump goreleaser/goreleaser-action from 4 to 5 (#1357)
Update to string (#1385) (#1390)
doc: improve documentation of BlockParams.MaxBytes (backport #1405) (#1410)
update language (backport #1263) (#1268)
build(deps): Bump styfle/cancel-workflow-action from 0.11.0 to 0.12.0 (#1445)
build(deps): Bump bufbuild/buf-setup-action from 1.26.1 to 1.27.0 (#1446)
build(deps): Bump pillow from 9.3.0 to 10.0.1 in /scripts/qa/reporting (backport #1493) (#1495)
build(deps): Bump bufbuild/buf-setup-action from 1.27.0 to 1.27.1 (#1520)
state: node bootstrap in v0.37.x (#1514)
indexer-respect-height-params-on-query (backport #1529) (#1543)
build(deps): Bump bufbuild/buf-setup-action from 1.27.1 to 1.27.2 (#1549)
Updates grpc and net dependencies to avoid https://pkg.go.dev/vuln/GO-2023-2153 and https://pkg.go.dev/vuln/GO-2023-2102 (#1590)
mempool: Add metric size of pool in bytes (backport #1512) (#1567)
build(deps): Bump bufbuild/buf-setup-action from 1.27.2 to 1.28.0 (#1601)
Backports #1558 and #1584 to 0.38.x (#1592) (#1611)
This commit makes the test be the same as in main
v0.37.x: Bump Go version to v1.21 (#1625)
Update SECURITY.md (backport #1626) (#1634)
Comment that feature only applies to v0 mempool (#1631)
Release v0.37.3 (#1640)
proto: Prepare for publishing v0.37.x protos to Buf registry (#1646)
proto: Update README (backport #1648) (#1652)
build(deps): Bump docker/build-push-action from 5.0.0 to 5.1.0 (#1656)
build(deps): Bump bufbuild/buf-setup-action from 1.28.0 to 1.28.1 (#1657)
v0.37.x: Restore minimum Go version (#1669)
mempool: add nop mempool (backport #1643) (#1681)
docs: various small improvements (part 2) (backport #1683) (#1686)
Do not block indefinitely on the semaphore (backport #1654) (#1689)
Release v0.37.4 (#1699)
types: validate Validator#Address field (backport #1715) (#1722)
fix: increase abci socket message size limit to 2GB (backport #1730) (#1746)
fix: Txs Validate (#1687) (#1752)
Update CODE_OF_CONDUCT.md (#1708) (#1767)
[e2e] Fixes prepareProposal not to return oversized set of transactions (backport #1756) (#1774)
Add test missing in #1687 (backport #1712) (#1760)
build(deps): Bump actions/setup-go from 4 to 5 (#1792)
build(deps): Bump actions/stale from 8 to 9 (#1793)
Introduce countAllSignatures in VerifyCommitLight & VerifyCommitLightTrusting (#1806)
Add changelog for #1749 (#1807) (#1820)
ci: fix failing golangci-lint action (#1833)
consensus: return last saved BeginBlock
build(deps): Bump actions/upload-artifact from 3 to 4 (#1840)
Updates go crypto package to v0.17.0 (backport #1859) (#1864)
Allow blocksync to not verify all signatures (backport #1858) (#1871)
docs: Fix Discord links in README (backport #1874) (#1895)
build(deps): Bump bufbuild/buf-setup-action from 1.28.1 to 1.29.0 (#2168)
build(deps): Bump slackapi/slack-github-action from 1.24.0 to 1.25.0 (#2167)
build(deps): Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 (#2166)
implementing missing interface
build(deps): Bump github.com/cloudflare/circl from 1.3.3 to 1.3.7 (backport #2253) (#2256)
feat(consensus): additional sanity checks for the size of proposed blocks (backport #1408) (#2140)
build(deps): Bump golangci/golangci-lint-action from 3 to 4 (#2297)
perf(internal/blocksync): avoid double-calling types.BlockFromProto (#2016)
perf(internal/blocksync): do not ValidateBlock twice (#2026)
docs: images not rendering properly in docs (backport #2331) (#2339)
add mergify
ci: fix failing golangci-lint action (#1833)
vulncheck
add the apphash error channel to the no-op mempool
add block sync metrics (backport from main)

PR checklist

[ ] Tests written/updated
[ ] Changelog entry added in .changelog (we use unclog to manage our changelog)
[ ] Updated relevant documentation (docs/ or spec/) and code comments

this is not a good way to do this.


PR for findByIdAndUpdate() solution for the PATCH request issue

This pull request makes the following changes

Fixes code-collabo/node-mongo#45

it shows the implementation of the find and update method of solving the patch request successful when ID does not exist issue

Ping @code-collabo/node-mongo-api-boilerplate-templates

@Ifycode I've just done the find and update solution for the ESM template.

@Ifycode I've just done the find and update solution for the CJS template.


Add http support to the spring-fat integration test

The spring-fat integration test starts a simple Spring Application with a single bean and this bean prints some logs to the stdout. If stdout contains required logs, we can say that test passes successfully. You can read more about the integration test right here.
Now we want make this test more "realistic". It would be nice to add http support to the application. The possible scenario might be:

Start Spring Application with some HTTP server (Spring Web?)
Make one http request to this server
Check the response status and content

Of course, we need to make jeo-maven-plugin:disassemble and jeo-maven-plugin:assemble before running the testing script described above.

@volodya-lombrozo can I take this issue?

@h1alexbel Sure, please, go ahead! Btw, I did something similar to this issue here. Hope it will help.


spotify-spicified: init

Motivation for this change
A little while back, spicetify-cli has been added to nixpkgs. However, Spicetify is build to edit the Spotify files, to theme/extend the application. To make this work with the immutable nature of nix, I made derivation in which the user can pass their config to the package. I personally used this for a while already in a personal repo (https://github.com/pietdevries94/spicetify-nix).
I also added the spicetify-themes package, to provide the most popular themes. I also noticed that the spicetify-cli package is missing some folders, as seen at the AUR (https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=spicetify-cli#n27)
I'm not entirely sure what the correct way forward is, but I thought it was most productive to make this PR, so I don't put in any more effort into the wrong direction.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

We probably want a derivation somewhere and a version number. Also the commits should be squashed together and follow the contributing guide.

I have squashed the commits. However, I have questions about the correct way to make a separate derivation and the version number, because this package is merely a modification of the Spotify derivation.
Do I have to make a derivation in which I copy everything over from Spotify and apply Spicetify-Cli to it?
And do I have to use the Spotify version as the version of the derivation, a combination of the Spotify and Spicetify version or an arbitrary 1.0?
I'm truly not sure about the first question, but I feel like the version should at least include the Spotify version

I tried
pkgs.spotify-spicetified.override {
  spotifyLaunchFlags = "--transparent-window-controls";
}

causes Nix to wait for spicetify to apply config.
It stucks at: building spotify-spicified-<IP_ADDRESS>6.g416cacf1 (installPhase): info Run "spicetify apply" to apply

I'm still patiently waiting for this! I hope we can get this working!

@pietdevries94 is it OK if I rebase this to another PR? I got everything working at https://github.com/danielphan2003/flk/blob/main/pkgs/applications/audio/spotify-spicetified/default.nix

This is already solved in the-argus/spicetify-nix

I will close this then and in case anyone want's get this upstream just open a new pr.


Fix telegram bot and add central config file

As #15 never actually ended up getting merged, then this is a follow up on that one, which brings a lot of improvements like improved code quality, faster execution and a telegram bot that not only actually works now (at least in my testing) but also uses some much needed formatting for the messages.
Besides this, it brings all the structural updates that #15 brought along (except that the telegram part actually has been tested this), which means that the commands for deployments have to be changed slightly, and there's a config file that need to be filled in. All of that is however reflected in the updated README, and it shouldn't take more than a few minutes to fix.
EDIT: Nearly forgot to mention, this PR also implements proper logging, such that future errors should be a lot easier to debug


Sync Style

Create a button in the style manager to sync the style of the the maps of the two sides

refer to #25

Done


B1500: Add measurement status to current/voltage parameters of B1517A SMU for spot measurements

dang! i accidentally pressed merged while i just wanted to update the branch. i'm very sorry. i'm gonna test this on the instrument tomorrow. in the meantime if you have any review comments let me know, i will fix them in another PR.

tested on the instrument and it does work. also the master branch build successfully. so the merge of this branch luckily actually went well :)


Create and Validate POC Metrics defined to validate with WHO

The goal is to create a lay out who the list of metrics defined by who will look like and be calculated.
From Previous https://github.com/liliana3186/sbv-world-health-org-metrics/issues/2 Create a Proof of Concept with the metrics we have define available to get.

I would like us here to do sketch what will be showing generally from the data we are able to reach and show to our Stakeholder to make sure we are in the right direction. This will give us a sense of the Visualizations to build out.

Added the metrics calculations, enhanced the fields we need  here


Request for visualization demo

Is there a visualization demo for image occlusion recovery? Want to see the image recovery effect of the model with high masked-up.

Hi @HaoqianSong.
Are you interested in a code/notebook to visualize the reconstruction of masked patches, as we do during training or is it something else ?
Renaud

Not for training, for interactive visualization demo, like  https://colab.research.google.com/github/facebookresearch/mae/blob/main/demo/mae_visualize.ipynb

Hi @HaoqianSong,
Sorry for the delay.
We have just updated the repository with a demonstration tool. You can run an interactive demo of CropMAE in the cloud using Colab Notebooks, or locally with the Notebook Demo.
Please don't hesitate to ask if you need anything else!


Projects in a solution folder are not unloaded on Install

Description
Projects in a solution folder does not appear to be unloaded when running Install.

Repro steps

Open a solution with a solution folder with projects in it.

Run Install.

Expected behavior
Projects to be unloaded when in a folder.
Actual behavior
Projects not unloaded when in a folder.
Known workarounds
Please provide a description of any known workarounds.
Related information

Win 7
Paket 3.1.7
.NET 4.5.1

Perhaps not very important, maybe just a cosmetic issue?

Is this related to the fact that unsaved files are not saved before Paket runs either?


Dead ?

Hi is this project dead ?

Unfortunately, I would say yes
I could not manage my free time to refactor this properly. So, in my opinion, this library has to be re-created from scratch again and maintained by a group of people, not only me.
I will make it read-only for now


DataFusion + Conbench Integration

Here's a minimal DataFusion + Conbench[1] proof of concept.
[1] https://github.com/conbench/conbench
A few notes (areas for improvement, caveats, etc):

Criterion results are in nanoseconds, but the smallest unit
Conbench currently speaks is seconds (because Conbench was initially
for macro not micro benchmarking). I suspect most places in Conbench
would work just fine if nanoseconds were passed in, but I need to
audit the code for any places that assume seconds if it isn't a
throughput benchmark.

If the Criterion benchmarks were named better, I could tag them
better in Conbench. For example, I suspect sqrt_20_12, sqrt_20_9,
sqrt_22_12, and sqrt_22_14 are parameterized variations of the same
benchmark, and if they were named something like "sqrt, foo=20,
bar=12", I could batch them together & tag their parameters so that
Conbench would automatically graph them in relation to each other. I
was sort of able to do this with the following benchmarks (because
there was a machine readable pattern). Anyhoo, that's easy enough to
do down the road as a last integration step, and it does appear from
the Criterion docs that they have their own recommendations for how to
do this.

window partition by, u64_narrow, aggregate functions
window partition by, u64_narrow, built-in functions
window partition by, u64_wide, aggregate functions
window partition by, u64_wide, built-in functions

While Criterion benchmarks can also measure throughput in some
cases, all the arrow-datafusion benchmarks were in elapsed time (not
sure about the arrow-rs benchmarks), so I didn't bother writing code
to support potential throughput results from
arrow-datafusion/arrow-rs, but we may need to revisit that.

We probably want to add some additional context, like the
arrow-rs/arrow-datafusion version, rust version, any compiler flags,
etc.

Thank you @dianaclarke  -- I will try and review this carefully over the next day or two. CC @andygrove  and @Dandandan

Thank you @dianaclarke -- I will try and review this carefully over the next day or two. CC @andygrove and @Dandandan

No rush & no pressure to merge this (or the sister arrow-rs pull request). I don't actually work on anything Arrow related anymore – just didn't want to leave you hanging. Cheers

If there are no objections, I plan to merge this PR next week so that I can proceed to the next step of integrating this with CI. The changes in this PR are self-contained so I think it is low risk to merge.

Thank you @dianaclarke , this is going to be a big productivity boost for us on all performance related PRs :)


This is using Gazebo headless on Arch Linux.

Nevermind, this only happens when you remove the ground in Gazebo. Don't ask... :smile:


I love it!!!

I really like this project. It's very good. But I have one comment, when I listen to a song, I don't know the name of the song that's playing now. So, if I could add the name of the song that is currently playing, I think it would be perfect, so if there is a good song, I can find it

Thank you! Glad to hear that you are enjoying Lofi FM 💜
Currently Youtube API doesn't have this support of getting the video fragment name, so I dont see  a solution for this right now. If you have found a fix feel free to make PR or comment about it.

图一是youtube 直播的画面,左上角有歌曲的名字.但是图二是你的播放页面并没有全屏显示出youtube的画面,你能解决后更新么?非常谢谢你.这样我就可以看到当时直播的歌曲了.非常抱歉的是,我是中国人,我的英语不太好,所以,请见谅

Yes, currently the background video size is bit scaled to hide the youtube controls. So the solution will be adding a button to disable that scaling so that you can see the video in original height and width. I will update the app during this weekend 👍

Thank you! 💯 :)

@jackey-jiang  Hi. i have added a toggle button to disable / enable the scaling. 👍 let me know if you found any issues


How to mock calls to DynamoDB

Where I have the following DynamoStore instance:
            const client = new AWS.DynamoDB({apiVersion: '2012-08-10', region: 'eu-west-2'});
            store = new DynamoStore<Event>(Event, client);

And the following code to run a query:
        const query = await this.store.query();
        query.params.ProjectionExpression = "#pk,#sk";
        const events = await query
            .wherePartitionKey(serviceId)
            .whereSortKey()
            .between(start, end)
            .execFetchAll();

I would like to write a test that exercises all of this code to ensure that the correct query is being made to AWS. We typically use the AWSMock library to do this using AWSMock.setSDKInstance(AWS);.
My issue is that for some reason, when using dynamo-easy the calls are still made to AWS. I can swap out the above dynamo-easy query with a raw call to Dynamo as follows:
        const res = dynamo.query({
            TableName: "test-table",
            ProjectionExpression: "#pk,#sk",
            ExpressionAttributeNames: {"#pk": "pk", "#sk": "sk"},
            ExpressionAttributeValues: {
                ":pk": {S: "4164"},
                ":sk": {S: "1554940800000"},
                ":sk_2": {S: "1555027199000"},
            },
            KeyConditionExpression: "#pk = :pk AND #sk BETWEEN :sk AND :sk_2"
        }).promise();

And the AWSMock library intercepts the call as expected. Can you please assist in what I am missing to make this work?

Of course, I find my own solution minutes after creating an issue here :(
My code was setting the mock expectation after I instantiated the DynamoStore class. By moving the mock expectations to before where DynamoStore is instatiated things started working as expected.

@frak good to hear it works for you


HIV is a subClass of cancer?!

Mondo term (ID Label)
MONDO_0022034 lentivirus infection
Bug/Typo/Error description
MONDO_0022034 lentivirus infection is asserted to be a subClass of MONDO_0006485 uterine carcinosarcoma
Therefore a query for
?s rdfs:subClassOf*  <http://purl.obolibrary.org/obo/MONDO_0004992>
reports that MONDO_0005109 HIV infectious disease is a subClass of MONDO_0004992 cancer

This superclass assertion comes from MESH:D016180. It does seem like a strange classification. @cmungall could we remove this SubClass Of assertion?

I believe this was fixed, I don't see this bug anymore. Please reopen if needed.


Implement PR merge command

Background
Initially this was just a small hack in my local hub repo. After having used it without any issues almost every day for the past two years, I thought I would open this PR for discussion to see if others are also interested.
Personally, I find it super convenient not having to open a browser/website to merge a PR (which I usually create from command line as well using hub pull-request).
Details
The GitHub API has offered a "merge" functionality for a long time:
PUT /repos/:owner/:repo/pulls/:pull_number/merge

See https://developer.github.com/v3/pulls/#merge-a-pull-request-merge-button
So far, this was not supported by hub.
I added a new hub pr merge <PR-NUMBER> command that takes the PR number and merges the PR (if the user has write access to the repo).
Note: This is the first PUT route - I had to add a PutJSON function to github/http.go.
Examples
$ hub pr list
      #1  Create README.md

# (Success case - PR 1 is merged, no output, exit code `0`)
$ hub pr merge 1

# (Error case, for example user has no write access)
$ hub pr merge 1
Error merging pull request: Not Found (HTTP 404)
Not Found

I also slightly updated the hub pr documentation to better reflect what it can do now.
$ hub
...
These GitHub commands are provided by hub:

...
   pr             Manage GitHub Pull Requests
...

$ hub merge
Usage: hub pr list [-s <STATE>] [-h <HEAD>] [-b <BASE>] [-o <SORT_KEY> [-^]] [-f <FORMAT>] [-L <LIMIT>]
       hub pr checkout <PR-NUMBER> [<BRANCH>]
       hub pr show [-uc] [-h <HEAD>]
       hub pr show [-uc] <PR-NUMBER>
       hub pr merge <PR-NUMBER>

Future ideas
Some ideas for the future (either as part of this PR or a follow up PR):

Improve error messages
Support different merge methods
Before the merge, see if all checks are configured and have passed - if not, abort merge
Allow "force" merging (if checks have not passed, see previous point)
Allow customizing the merge commit message
Add an option to automatically delete the PR head branch after successful merge

Count me as a :+1:
I'm not too worried about the confusion, pr merge is what happens when you click merge on a pr. But merge is just... a local merge. I think it maps well, though the docs should clarify how each works.

This looks great.
It would be also worth thinking about rebase and squash actions.
Hypothetical those could be ‘hub pr rebase/squash N’ or an alternative ‘hub pr accept —merge/squash/rebase’

I also see no confusing between hub merge and hub pr merge. I find it much harder  to comprehend that there is hub pull-request instead of hub pr create...
However, what I am missing in my current workflow, is a quick way to

merge the PR
delete the feature branch on GitHub
delete the feature branch locally (i.e., run git fetch --prune and git branch -d)

(1) and (2) can currently be done using the browser; (1) can be done with your pr merge feature, and (2) is planned for the future. So this is a feature request for your feature PR: please also think of (3) when implementing (2). :-)

@sbeyer - do you find that when PRs are merged, git branch -d works for you?  I find git never knows it's there (perhaps because we do squash merges), and thus git branch -D is required... been trying to find a sane way to sanity-check that in a wrapper script but have failed.

do you find that when PRs are merged, git branch -d works for you?

It does, but I don't use squash merges. Git does not track branch history, so it does not know when a branch is squashed. I btw never got to know why someone wants to use squash merges in general (I can think of a few use cases though... well, that's all off-topic, sorry). However, the example of squash merges (which should probably also be supported by pr merge) is a good example where a "delete remote and local branch" option makes even more sense (and a "delete only remote branch" also makes sense because the original commits are kept).

I btw never got to know why someone wants to use squash merges in general

I know this is way off topic, but since you asked, and it's a valid question...
On PRs with a ton of back and forth, I don't want the history of "fixed typo" "changed casing" "address code review comments" "address code review comments" "code review" "more fixes"
It's messy, makes it hard to look through history, etc. So you have two options, you can either ask people to clean up their history before you merge and force-push and then merge (which usually a ff-merge), or you can squash-merge, which is really convenient.
This is the basic model of Phabricator (from Facebook), btw, except that you always amend, and phabricator keeps track of the delta between each time you update the PR (they call them 'diffs'), so the final history in the git repo is morally the same as a squash merge, but the various revisions of that "diff" are all also discoverable (this would be the same as if you GitHub kept the history of all force-pushes and you could go back through them all).
There's pros-and-cons each way, no way is "the right way", but that's the pros of that model and why people (at least most people I know), use it (especially if they came from google or facebook, both of which use that model).

@jetersen - You're right it does seem to be "sticky". So I agree it should force a clear choice with -- options to bypass prompts.
@friederbluemle - that work for yoU?

if anything it should follow the design of https://github.com/cli/cli/pull/873

Thank you @friederbluemle for the initial implementation and everyone else who contributed to the discussion!
The feature is now merged:
gh pr merge 12  # merge PR #12
gh pr merge --squash 12
gh pr merge --rebase 12
gh pr merge --delete-branch 12  # deletes head branch when done
gh pr merge --head-sha=<COMMIT-SHA> 12  # verifies head branch state
gh pr merge 12 -m "override merge commit title"

I have decided not to pick a merge method automatically. If people feel strongly about the merge method being --squash or --rebase on repositories that enforce linear history, you may open an issue or submit a pull request, but for now I expect the caller of pr merge to explicitly pass in a merge method as desired.

Thank you @mislav :+1:

Thanks @mislav ! One question, you mention:

gh pr merge 12  # merge PR #12
but also say:
I have decided not to pick a merge method automatically.
So what happens when you don't specify one which you show is possible with that command.

I would assume it automatically picks the default (i.e. a regular merge commit, as documented here):

Based on some testing I did a couple of months ago, you'd get an error message if the default method was disabled for that repo via the settings page on GitHub. You'd then need to explicitly declare either --squash or --rebase.

@friederbluemle That's right; thank you! /cc @jaymzh
I realize that the default behavior is not ideal for repositories where merges are disallowed, but maintainers in those repositories already know their preferred merge method and it shouldn't be hard for them to add the flag.


is there a way to do nightly / unstable releases?

Affected Packages
Idk
Problem
I may not be ready for a release, but i may want folks to be able to test out changes without manually doing pre-reloason
Proposed solution
Provide an unstable option in config. When true, every merge is published (provided the changesets declare changes) with the git sha appended to the version. This uses the 'unstable' tag on npm, and does not create github releasen, nor remove the changeset files

yeah, for unstable releases, I wouldn't care about noting changelog entries -- I (and I'm sure a good number of folks) review their changelog / changesets before actual release, proofread, etc etc -- all unneeded for unstable

I would also really like this feature. When the changeset publish PR opens (and each time it updates) an unstable version should be published.
Currently to do integration testing when multiple changesets come together we have to manually publish versions or use workarounds like links and yalc.

For now we've implemented this ourselves with a github action
name: Unstable/nightly release

on:
  push:
    branches: [main]

jobs:
  unstable-release:
    name: Unstable/nightly release
    runs-on: ubuntu-20.04
    steps:
      - uses: actions/checkout@v3
      - uses<EMAIL_ADDRESS>
      - name: Use Node.js 16.14.0
        uses: actions/setup-node@v3
        with:
          node-version: 16.14.0
          cache: "pnpm"
          node-version-file: ".nvmrc"
          registry-url: "https://registry.npmjs.org/"

      - name: Install dependencies
        run: pnpm install
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

      - name: Generate unstable changeset
        run: pnpm changeset version --snapshot next

      - name: Build packages
        run: pnpm build

      - name: Publish unstable packages
        run: pnpm changeset publish --tag next

@jake-slashid
Is this working the way you expect?
I guess your package versions are of the format 0.0.0-next-YYYYMMDDHHMMSS


How will I create migrations for the admin tables at the same time when I use voyager admin to create tables (I mean the database tab)?

Laravel Version: 5.4.16
Voyager Version: 0.11.9
PHP Version: 7.1.2
Database Driver & Version:

Description:
Hi I have an issue I am creating the tables for the admin panel using the database tab in the voyager but I notice that migrations have not been created for the tables. Please assist me how will I create migrations at the same time when I create table using the voyager admin panel.
Now the issue is when I store the project on git and some one takes the clone he does not get the migrations for the admin panel as it was not created. Please assist , thank you.
Steps To Reproduce:

Voyager does not currently support creating migrations.  It's a feature we're working on.  For now, you'll have to create the migrations manually.


[Bug]: Function call execution does not respect the work_dir

Describe the bug
Even if work_dir is specified in the code execution config, function call execution does not save files to it.
This prevents building applications that can use function calling properly.
Steps to reproduce
Run the following code, which I created by register one function in test/twoagents.py:
from autogen import AssistantAgent, UserProxyAgent, config_list_from_json

def download_file(url: str, file_path: str, headers: dict = None) -> str:
    """Download a file from a URL and save it to a file path. Return the file path."""
    import requests

    if headers is None:
        headers = {}
    response = requests.get(url, headers=headers)
    with open(file_path, "wb") as file:
        file.write(response.content)
    return file_path

config_list = config_list_from_json(env_or_file="OAI_CONFIG_LIST")
assistant = AssistantAgent("assistant", llm_config={"config_list": config_list})
user_proxy = UserProxyAgent(
    "user_proxy", code_execution_config={"work_dir": "coding", "use_docker": False},
    human_input_mode="NEVER",
    is_termination_msg=lambda msg: msg.get("content") and "TERMINATE" in msg.get("content", "")
)

assistant.register_for_llm(description="Download a file from a URL and save it to a file path.")(download_file)
user_proxy.register_for_execution()(download_file)

user_proxy.initiate_chat(assistant, message="Save the following paper https://arxiv.org/pdf/2308.08155.pdf to disk as wu-arxiv-2023.pdf")

It runs and gives the following output

But as a user, I'd expect the file wu-arxiv-2023.pdf to be saved in dir called coding, but instead it will save it in the cwd as shown below:

@ekzhu @jackgerrits @sonichi

It seems that execute_function does not even use docker or parameters from code_execution_config?
https://github.com/microsoft/autogen/blob/4d0d486115a5445737f7d9937c97ce4453ea15b4/autogen/agentchat/conversable_agent.py#L1959-L2012

Having the same issue

@Noir97 code execution and function executions are separate, so in function execution it is not going to use the work_dir inside code_execution_config. Take a look at these tutorial chapters.

Tool use: https://microsoft.github.io/autogen/docs/tutorial/tool-use
Code executor: https://microsoft.github.io/autogen/docs/tutorial/code-executors
User defined functions: https://microsoft.github.io/autogen/docs/topics/code-execution/user-defined-functions

@gagb @jackgerrits perhaps we can close this issue as "won't fix"?

I think any func exec should respect constraints set in the executor, but I am not blocked by this in lieu of udfs. So I am fine with this!

I agree with @gagb assessment that it should save to work_dir.  That given can we close this issue since @gagb  is not blocked and it looks like it is something we don't want to fix now.


Typha causing APIServer crashes on larger Openshift clusters

On larger Openshift clusters: restarts of the typha operator cause really large memory spikes that result in the APIServer getting out of memory events and effects downstream components. In my cluster I currently have
12374 pods
225 nodes
3314 services
kubectl get pods --all-namespaces -o wide --no-headers  | wc -l
   13274

   kubectl get nodes --no-headers | wc -l
     225

kubectl get services --all-namespaces --no-headers | wc -l
    3314

Note we have a total of 3 typha pods in the cluster
kubectl get pods -n calico-system -l k8s-app=calico-typha
NAME                           READY   STATUS    RESTARTS   AGE
calico-typha-b66bc54df-44zk7   1/1     Running   0          28m
calico-typha-b66bc54df-shh4f   1/1     Running   0          28m
calico-typha-b66bc54df-vl7jp   1/1     Running   0          28m

Posting the associated graphs that show the behavior when restarting typha (I can replicate by simply doing kubectl delete pods -n calico-system -l k8s-app=calico-typha) and waiting some time

At the time of the restart the kube-apiserver memory doubles after the restart of the pods (goes up by 20+Gigs).
The maximum size of the backing database if 4GB. It seems odd that on startup of typha it would result in over 20+GB of utilization when it's meant to be a single "pool" of connecting to the datastore for all the calico-node pods.
Expected Behavior

Memory usage of the openshift apiserver on a restart of typha should not double. Typha should be able to download the data it needs in a controlled fashion that doesn't cause the APIServer to drastically change from a memory perspective.
Current Behavior
At startup of typha pods memory usage drastically increases for the kube-apiserver hosting the cluster (doubles).

Possible Solution
Potentially try and rate limit data download for larger clusters? Maybe look at watch optimizations?

Steps to Reproduce (for bugs)

Get large cluster. The scaling characteristics I put above are within Kubernetes maximums  and can be utilized for reference
Restart typha pods
Watch memory of kube-apiserver while the calico-typha pods are starting up (takes about 10 minutes in my monitoring for results to be shown.)

Context

This issue affects the availability of the kube-apiserver. It can lead to slower processing of other workloads, drastically slower kubectl request processing (to the point where the requests time out), and failure of other control plane components like scheduling of pods, failover of pods, etc. This is due to the fact that calico-typha is consuming all the resources from the kube-apiserver
Your Environment

Operating System and version:
Link to your project (optional):

Cluster: Openshift 4.7.19 (4.7.19_1526_openshift IBM Cloud Openshift cluster)
Details about the scale of the cluster are listed at the top of the issue

I would suggest posting this to github.com/projectcalico/calico as what you are reporting is not something that is special to the operator. If you think it is particularly operator related could you please explain why you believe so.
You should also include the version you are using in your report.
How bad is the utilization if you restart only one typha pod? We wouldn't ever expect someone to restart all the typha pods simultaneously and would suggest not doing that for high availability purposes.

It seems odd that on startup of typha it would result in over 20+GB of utilization when it's meant to be a single "pool" of connecting to the datastore for all the calico-node pods.
Just imagine if you didn't have typha to fan out the information to all the nodes.

How many apiserver instances are your running? Maybe you need to scale the number of master nodes up.

Could also be related to this:
https://github.com/projectcalico/typha/issues/492

I’ll move to typha repo

I’ll also gather the data on restarting one but I don’t think the startup process should be causing this large of a spike under any circumstances. Or if it is would be interested in what the expected load characteristics are

I'm closing this issue since https://github.com/projectcalico/typha/issues/637 has been created.


.ci/lib.sh: install_yq fails on ppc64le

Hi @nitkon - it would be helpful (both here and in future) if you could put more detail into issues (error messages, log extracts, etc) so that we can all understand a bit more about them :)


swarm spans are uint64, but io.Writer and Seek methods use int64

Swarm spans use uint64, but the writers that we use and the native golang interfaces use int64 for reads and seeks.
It is not entirely clear on how this affect the binary and how seeks and reads will be affected beyond the int64 upper bound. It might also be a sort of attack vector, but needs further examination.
related to #619

@acud, what is the status of this?


ISSUE-6 Fix line endings in org.spdx.rdfparser package

Pull request to continue towards resolving #6 - as there are hundreds of files in the repository, of which all appear affected, I thought a good way to break it into manageable chunks would be to pick a package.
This resolves the line endings issue in the org.spdx.rdfparser package. There are no other changes in this pull request.
All of the files appear modified on every line due to the nature of the issue.
My contributions are licensed under the Apache 2.0 license as required for contributions to this project.

Merged into git.spdx.org - will be updated on github next time it syncs


CI: flaky compilation

Description
Sometimes compilation on CI fails due to compiler bugs.
[2020-06-01T20:07:22.772Z] /usr/local/bin/ccache /usr/bin/g++-7  -DDMLC_CORE_USE_CMAKE -DDMLC_LOG_FATAL_THROW=1 -DDMLC_LOG_STACK_TRACE_SIZE=0 -DDMLC_MODERN_THREAD_LOCAL=0 -DDMLC_STRICT_CXX11 -DDMLC_USE_CXX11 -DDMLC_USE_CXX11=1 -DDMLC_USE_CXX14 -DMSHADOW_INT64_TENSOR_SIZE=0 -DMSHADOW_IN_CXX11 -DMSHADOW_USE_CBLAS=0 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_MKL=1 -DMSHADOW_USE_SSE -DMXNET_USE_BLAS_MKL=1 -DMXNET_USE_LAPACK=1 -DMXNET_USE_LIBJPEG_TURBO=0 -DMXNET_USE_OPENCV=1 -DMXNET_USE_OPENMP=1 -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_SIGNAL_HANDLER=1 -DMXNET_USE_TVM_OP=1 -D__USE_XOPEN2K8 -Dmxnet_EXPORTS -I/work/mxnet/include -I/work/mxnet/src -I/work/mxnet/3rdparty/nvidia_cub -I/work/mxnet/3rdparty/tvm/nnvm/include -I/work/mxnet/3rdparty/tvm/include -I/work/mxnet/3rdparty/dmlc-core/include -I/work/mxnet/3rdparty/dlpack/include -I/work/mxnet/3rdparty/mshadow -I3rdparty/dmlc-core/include -isystem /opt/intel/mkl/include -isystem /usr/include/opencv -D_GLIBCXX_ASSERTIONS  -Wall -Wno-sign-compare -O3 -g -fopenmp -O2 -g -DNDEBUG -fPIC   -Werror -Wno-error=unused-variable --coverage -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -mf16c -std=gnu++1z -MD -MT CMakeFiles/mxnet.dir/src/operator/instance_norm.cc.o -MF CMakeFiles/mxnet.dir/src/operator/instance_norm.cc.o.d -o CMakeFiles/mxnet.dir/src/operator/instance_norm.cc.o -c /work/mxnet/src/operator/instance_norm.cc

[2020-06-01T20:07:22.772Z] /tmp/ccT4hWrG.s: Assembler messages:

[2020-06-01T20:07:22.772Z] /tmp/ccT4hWrG.s:566393: Internal error in stringer at ../../gas/read.c:5521.

[2020-06-01T20:07:22.772Z] Please report this bug.

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-18455/1/pipeline/
We may reduce the number of occurrences by switching to a compiler version with stable c++17 support on the unix tests (and only test gcc7 on centos tests)

Is this still occurring?

There was no change in the compilers, but possibly change in mxnet code avoids the problem. The problem wasn't frequent before, so it's hard to tell.
@ChaiBapchya can you help add the Internal error in regex to the CI Lambda Check? Or do we have another easy way to filter all CI logs for the prevalence of this issue?


Difference in connecting to LDAP server with version 5.5 and 5.6

I get a difference in connecting to my LDAP server with ldap2pg version 5.5 and 5.6
In version 5.5 everything goes well.
With version 5.5, I see in the log file:
Connecting to LDAP server ldaps://ldapserver.lan
Trying simple bind.
Doing: ldapwhoami -x -D CN=......................
Running in dry mode. Postgres will be untouched
Connecting to Postgres libpq default.
In version 5.6, I see in the log file:
Connecting to LDAP server ldaps://ldapserver.lan
Trying SASL GSSAPI auth.
Doing: ldapwhoami -Y GSSAPI
Failed to connect to LDAP: {'info': SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available (default cache: KEYRING:persistent:512))', 'desc': 'Local error'}
Why is version 5.6 trying with "SASL GSSAPI auth" and 5.5 with "simple bind"?
What is wrong and how can I solve this issue?
With regards,
Herman Pool

Hi @HermanPool , thanks for reaching.
After the changelog at https://ldap2pg.readthedocs.io/en/latest/changelog/#ldap2pg-56, I guess that you have something in /etc/openldap/ldap.conf. What tells ldap2pg --verbose about this ?
Regards

Closing stale issue.


AVRO-9262: Configure dependency updates via dependabot

This has to be enabled by INFRA too.

LGTM!  Thanks!

A bit late to the party here, but kudo's for making this happen! 👍

I am thinking about the big amount of pending PRs to merge that we need now some sort of plugin + whitelist of dependencies to auto-upgrade if tests are green, but maybe worth discussing in the ML


add string·elem_ords

spec: https://github.com/google/starlark-go/blob/master/doc/spec.md#string·elem_ords
$ starlark-go -c "''.elem_ords()"

(exit 0)

$ starlark-rust -c "''.elem_ords()"
error[CV00]: Cannot .elem_ords on type string
 --> [command flag]:1:1
  |
1 | ''.elem_ords()
  | ^^^^^^^^^^^^ .elem_ords not supported for type string

(exit 2)

As far as I can tell, starlark-rust's current implementation of elems does what the spec says elem_ords should do.
elems itself is interestingly specified... Rust doesn't really have a native concept of a byte string absent a charset in the way that Go does, so there isn't a particularly natural implementation for elems...

Probably the spec should say something about this, but I’m guessing the answer is UTF-8. cc @alandonovan

There is consensus that it is infeasible to specify the string encoding in a way that does not impose an intolerable performance penalty on at least one implementation, so each impl is free to define the element type of string as it wishes. Starlark-Java's strings are sequences of UTF-16 codes, Starlark-Go's strings are sequences of UTF-16 codes, and judging from the test suite, Starlark-Rust's strings are sequences of Unicode code points. (A poor choice from a performance perspective, but a legal one.)  All of the string operations are defined in terms of elements, not bytes/chars/codepoints.  The elem_ords method for Starlark-Rust should return a sequence of code points as integers.

The elem_ords method for Starlark-Rust should return a sequence of code points as integers.

Yeah, elem_ords is easy - bytes can be represented as numbers in an obvious way; the question here is about elems - it appears to return a string representation of arbitrary single bytes. In the Go implementation, these end up as strings like \xe4, but because rust treats strings much more as utf-8 and much less like bytes, there isn't an obvious string-ified version of a byte. We could use an escaped string \\xe4, but that is more of a debug representation - if you were to join it with some other bytes, there would be slashes, rather than code-points merging into multi-byte characters.

I think a third option would be to lazily compute and cache the char array per string only if it is requested. I don't know how common it is for your use case, but for mine, string manipulation from the starlark side is pretty rare.


[FR] Support Markdown outputs

From nodes, or of boards?
In our kits package, I ended up writing utility functions to output boards to MD, with the mermaid and/or json.

@dglazkov We don't currently emit a markdown behavior anywhere, but if we do I think I should be able to wire this one in pretty quickly.

I should totally add it in at least one place ...

Tour guide writer now has format: "markdown"!

I also just realized something related: most of the time, LLM output expects to render in whitespace-preserving way. Like this poem:
A board of plastic, holes in rows,
For circuits temporary, as everyone knows.
No soldering required, just wires to poke,
A playground for electrons, where circuits awoke.

LEDs blink, resistors resist,
Capacitors charge, transistors persist.
A breadboard's a canvas, a playground to test,
Ideas take form, put to the quest.

From simple circuits to complex designs,
A breadboard's the place where creativity aligns.
So grab your wires, your components too,
And build your dreams, make them come true.

However, we render as if it's just HTML content. WDYT about changing the default to white-space: pre or something like that? I bet that would go a long way.


dts: usbserial: Fix USB serial console device name

Fixes DT device name for USB serial console.
Signed-off-by: Andrei Emeltchenko<EMAIL_ADDRESS>

Codecov Report

Merging #14199 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #14199      +/-   ##
==========================================
- Coverage   52.68%   52.67%   -0.01%
==========================================
  Files         307      307
  Lines       45473    45473
  Branches    10530    10530
==========================================
- Hits        23956    23955       -1
  Misses      16640    16640
- Partials     4877     4878       +1

Impacted Files
Coverage Δ

arch/posix/core/posix_core.c
90.9% <0%> (-1.02%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 10bdc2a...18ba9a5. Read the comment docs.


Allow multiple selection

Hi,
It would be nice if one could select multiple countries. Would you think about doing so in a near future?
Thanks, Louis

I have an idea how this might work well. I'll add it to my to do list.

Hi @LouisBorlee! I was working on this a little today. What is your specific use-case or use-cases for multiple selection. I would help me with the API design.

Hi!
I would like to use it as a "Select your favorites countries".
Maybe you can pass an argument that would tell your controller that multiple selection is allowed.
Something like planetViewController.isMultipleSelectionEnabled = true that would add checkmarks when tapped instead of a dismiss with some bar button in the navigation bar to do it.
If you need anything else, don't hesitate to ask!
Cordially, Louis.


Allow to specify how automatic public path should be determined

Bug report
When building module script and my bundle contains a relative import statement then bundled code contains import.meta.url to detect public path. The problem is that the import keyword is supported from chrome >64 but support for javascript modules via script tag was added in chrome >61. I use module/nomodule approach to load es5 in old browsers and es6 in new browsers but there are some versions such as chrome 61, 62, 63 which are not covered because of this unsupported import keyword.

What is the current behavior?
If the current behavior is a bug, please provide the steps to reproduce.
Currently, there are two ways to automatically detect public path:

 via import.meta.url - when script type is set to module

 via document.currentScript.src

What is the expected behavior?
Consider adding more options to a public path something like:
  output: {
    publicPath: "auto/current-script" | "auto/import-meta",
  },

Other relevant information:
webpack version:  latest
Node.js version:  latest
Operating System: any
Additional tools: N/A

Actually just realized that it cannot be used like that because document.currentScript is null when loaded as <script type="module">` I guess there is no solution for this problem and I will have to learn to live with that. 😢


Fix typos in SPEC.md; use full isolator names in headers

Squash please.

@jonboulle added one more commit: e559b0e
squashing...

@jonboulle merge away

Thank you for the fix.


Filter by patent type and filing date range

patent type being utility, plant or design
filing dates being 20 years ago for utility and plant and 15 years ago for design
or if term grand is available then filter by 20 years etc

Here is a search using the legacy version that searches patents filed in the year 2004 and are of the types "Utility" and "Plant" only.
https://api.patentsview.org/patents/query?q={"_and":[{"_gte":{"app_date":"2004-01-04"}},{"_lte":{"app_date":"2004-12-31"}},{"_or":[{"patent_type":"Plant"},{"patent_type":"Utility"}]}]}


[Net] Rename StreamPeerSSL to StreamPeerTLS.

SSL has been deprectated almost 10 years ago.

Should there be a compatibility binding like we have in register_scene_types.cpp or is that not relevant/possible for core types?

Looks like the only core remapping we have in register_scene_types.cpp is for s/AStar/AStar3D/.
But I'm not sure it's actually useful, both AStar3D and StreamPeerTLS are neither Resource nor Node but extend RefCounted directly, so I don't see how the compat remapping would be used (for nodes and resources it's used to allow opening scenes and resource files from 3.x and remap the types automatically).

Thanks!

Should there be a compatibility binding like we have in register_scene_types.cpp or is that not relevant/possible for core types?

I've added the class rename to the converter, I think that should be enough.

The ssl variables in the changed code (and other related ones like use_ssl) could be changed too, but it's not critical (could also be done in the future as a general code quality pass).

Sure, I'll make another PR :)


[Linux] boost::process:: child ::wait_until is not thread-safe

Hi! I faced with the problem that my process hangs up while it uses wait_until() of child::process. So I watched boost code and find that boost::process::child::wait_until is not thread-safe

struct signal_interceptor_t  is used for setting custom reaction on signal with using system ::signal. So handler_func is a new handler and sigchld_handler is setting for previous handler
The problem is that signal handler - atribut of the process, not thread, so if the next thread call ::signal then sigchld_handler is setting for handler_func
Which thread will be used for handle signal is undefined. So thread with  sigchld_handler == handler_func can be used! And this cause infinite loop
You can face with the problem in this code, which uses problem part of wait_until. (On my system signals are handled by main thread). Also attach out of the programm
bug_wait_until.cpp.txt
out.txt

So can you update your documentation with this note please?

Close this issue and open in right place


A few questions about this project

Hello there, I recently stumbled across this project looking for new repositories to contribute to. Seeming as I tried to create a platformer in unity a while back, this one drew me in.
I have looked around trying to find some documentation or instructions on how to use this project but I am currently unable to find any. Has there been any created yet, or is this more currently at a personal state?
I notice that this project uses 2 spaces instead of the standard 1 tab (tabs allow individuals to define how wide they appear in their own ide). Is there any reason you chose 2 spaces instead of tabs? would you be against converting the project to tabs (if you wish, I can create a PR converting it to tabs)?
Now I am not sure if this problem exists with Unity 5.5.1f1 (which is what the project currently supports), but I am using 5.5.2f1 and an error comes up when opening the project saying that IntegrationTest does not exist for Test/IntegrationTestDuration.cs on line 23.
In order to fix this (at least for 5.5.2f1) the UnityTestTools extension is required.
Was this an issue in 5.5.1f1? If so, would you be up for including the extension (without the example files of course) in the projrect? Or did you intentionally exclude the extension (for maybe, say, low project size or something)?
If that last question regarding UnityTestTools was answered somewhere that I overlooked (such as the instructions or documentations that i was asking about previously), I apologize.

Project status is: in-progress-stable, I'm developing a game with it.
Expect some breaking changes, still no semver.
There are at least two more projects in progress, people ask me questions by email.
About documentation / collaboration.
I created many issues for improvement or caveats, and things I don't really need but it could be cool...
There is some documentation at: https://github.com/llafuente/unity-platformer/wiki
And the code is rather well documented and not too complex to read. I don't over-engineering things (or try at least).
tabs vs spaces... I prefer spaces myself because I use vi sometimes so all my editors are 'space-only'...
But If you find something like clang-format for c#
I would be happy to format the code with it. But should be automatic like in my c projects: https://github.com/llafuente/flang/blob/master/format.sh
UnityTestTools is requied it's in the readme :roll_eyes:  :+1:
I will dedicate some time to documentation.
I think videos are far easier to understand but I'm no english native... will see...

I have created a "project" to track things i want for v1
https://github.com/llafuente/unity-platformer/projects/1
I may have time next week to start working on it and finish some documentation work.

Hey that's fine man I just seen your other comment it makes more sense. Have you considered maybe including an example project with the source? That may be an easier alternative. I may be able to help you out with documentation and code comments later on.

Documentation updated. http://llafuente.github.io/unity-platformer/
I will deprecate and remove the wiki soon, I moved everything to API-doc (Assets/Docs, maybe not the best place... will see)
No much time lately but I'm trying!

As dumb as this is, I actually just lately took the time to read the about for this, and realized it's based off of a tutorial series. Likely I should have watched it before jumping in and reading the code, I think I was bored the day I was skimming the code. I'm going to watch the YouTube turorial series before I play with this or ask anymore questions.  That said, you don't need to rush yourself with documentation. What exactly do you mean by api-doc? Are you referring to the in code summary tags?


pyupgrade: 2.38.1 -> 3.0.0

Description of changes
Update to latest upstream release 3.0.0

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 194993 run on x86_64-linux 1

2 packages built:

pyupgrade (python310Packages.pyupgrade)
python39Packages.pyupgrade


chore(pkg/wireguard): remove duplicate link syncing, and cleanup

Fixes: #166

LGTM!

Will try and test this before merging, thank you.


Add link to "submission under review" email

@nathanshammah is requesting that we add a link to the submission on Metriq, in the "submission under review" email. He also suggested we consider italics or bold for the title, but this isn't handled by the most rudimentary or oldest SMTP versions for email, though it might be done anyway.
@vprusso Adding "Architecture" back end and views is going to consume most of my bandwidth, this week. Could you also take a look at this issue, please?

Closed in https://github.com/unitaryfund/metriq-api/pull/180


fix: screenpipe support in china

This PR fixes hugging face being blocked in China.
User can specify env variable HF_ENDPOINT=https://hf-mirror.com for CLI to use China mirror
type of change

[x] bug fix (non-breaking change which fixes an issue)

/claim #341
/closes #341

@neo773 i think we download stuff from other places like github
#383 -> prob this would need a hack for china too (e.g. host some ffmpeg build on chinese-friendly data center), but can do later

Github is not banned in China only hugging-face

Tested the ffmpeg URL with this website, it's not blocked https://www.comparitech.com/privacy-security-tools/blockedinchina/

/approve
thank you :)


SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame

This lines yields a warning:
SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame
See https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

Solved in #184


SPMT tool settings for max file size upload

[Enter feedback here]
Hello team,
is there any feature planned for setting maximum single file upload limit for migration to OneDrive.
With current update from OneDrive, it now allows 100GB of file upload, while our requirement is to not migrate files greater than 15GB.
Thanks and regards,
Rupali Kothawade
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: dad86450-d9ac-4cc4-fc75-63f8b285c298
Version Independent ID: d1e8c85a-b365-78c0-be1a-5a03ee082491
Content: Available settings in SharePoint Migration Tool - Migrate to SharePoint and OneDrive
Content Source: migration/SPMT-settings.md
Service: sharepoint-online
GitHub Login: @JoanneHendrickson
Microsoft Alias: jhendr

@Rupalikothawade Thank you for submitting feedback.
I think the best way forward is if you open a service ticket in your tenant so this can get resolved ASAP. Based on the outcome let me know if it is something that can be called out in the docs.
Please follow this link to contact support for business products:https://docs.microsoft.com/office365/admin/contact-support-for-business-products
Please keep us posted here on the resolution so we can feed whatever information you discover into the content.


Slide does not keep state when shown as cloned version when loop: true

Issue description:
When a user clicks on a slide, I am toggling a class that shows the slide with a "selected" style using a modifier class on the slide. This works fine until the slider loops back around, in which case the style on the selected slide is completely absent. It's as if it was never selected. This is specifically the case if the slide has the following classes:
tns-item tns-slide-cloned tns-slide-active
It seems that slides that are cloned and active do not update to have any of the state changes of the original (non-cloned) slide that was selected. Is there anyway to preserve the selected state changes from the originally selected slide when the slide is presented as cloned and active?
Demo link/slider setting:
const slider = tns({
  container: $carousel,
  items: 2,
  gutter: 6,
  slideBy: 'page',
  mouseDrag: true,
  responsive: {
    1200: {
      items: 3,
      mouseDrag: false,
    }
  }
});

I've created a demo here. If you click the first slide and then click "Next" 4 times, you will see that the slide no longer has the selected state. I've noticed you have to click "Run" at the top corner of JSFiddle to get the demo to work.
https://jsfiddle.net/brianecook/7qxhkbng/1/
Tiny-slider version:   ^2.9.4
Browser name && version:  Firefox
OS name && version:  Mac Ventura 13.4

Hi, were you able to solve the issue? I am also facing it currently.


feat: Use Tree of Thought paradigm to produce more relevant responses.

Is your feature request related to a problem? Please describe.
Sometimes, okay, a lot of times, the response is unsatisfactory. Sometimes a search is needed, but the prompt didn't include the magic phrase.
Sometimes, you get an "As an AI model" deflection.
Describe the solution you'd like
The major takeaway I got from the Tree of Thought paper is the application of a spam and prune method of solution seeking.
So, let's run a query by all personalities, then pick a winner, or possibly, compile the results.
It might be worth exploring also whether increasing temperature and k would also yield a diverse enough set of potential solutions.
A risk to this approach is that the technique is ill suited to the query and not enough diversity of responses is generated, resulting essentially in wasted spend. A countermeasure to this risk could be first to do a naive talk to all personalities approach followed by high-temp and high-k spam.
As such, the evaluating prompt node(s) should evaluate
a) the best response of the set (including a compilation of that same set)
b) whether additional passes should be called for. For now, let's just have the bot say as much as opposed to building it out.

So #42 emperically improved the quality of responses.
However, it's not really a tree of thought until the evaluator is itself maintaining the tree of thoughts and can traverse the branches in order to produce the desired output.
Worth further exploration.


[WEB-990] Support Next.js server actions

From SyncLinear.com | WEB-990

thx for the precision, it was confusing for many users.


Small improvements and absolute percent error for the Julia Monte Carlo

Therefore attacking #324

That was quick :)


createAccount example

Is it possible to get a little createAccount example.
I read "CreateAccount SOAP Command" doc in zimbra docs, but still have some trouble to apllicate it.
Thank you very much.

Ok, finally I found how to did it :
$create_account_res =<EMAIL_ADDRESS>"secret");
Now the problems are :

How to check if user already exists? Or how to intercept an Exception that will be thrown because the user exists.
How to print (or analyse) the return of  command, i.e. $create_account_res

I saw that $create_account_res has a type of object.
It was an Xml that was converted into Object (there is a method toObject() for this).
But I have somedifficulties to work with this object... how to do it?
Or is it possible to get directly an Xml and not an object ?

I will try to help for you with your first question.
You can try to add, if it's exists, already exists will be return. Or if you want to only check, you can try to get account details with GetAccount. if it's exists account details will return. if it's not, not found will be return.
I hope understand your question.

Thank you kazimc.
But my issue is more concern about lack of documentation and/or examples for this project. And it's complicated by fact that I have not experience/knowledge in php development.
Probably for php developers it should seem obvious how to use this library, but not for me.
So my question was most about code example, not the algorithm.
Now, to get the things more concrete. This code is very ugly, but is it correct on the idea level?
I know I do twice almost the same check. But it better check twice than zero...
<EMAIL_ADDRESS>$account = new \Zimbra\Struct\AccountSelector(\Zimbra\Enum\AccountBy::NAME(), $account_name);
try {
     $accountInfo = $api->getAccountInfo($account);

     echo "Some problem!
";
     echo "This account is probably exist already!
";
     echo $accountInfo->name."
";
     print_r($accountInfo);

     exit(1);
} catch (GuzzleHttp\Exception\ServerException $e) {
/*
    Ok, server throw an Exception, so account probably does not exist.
    echo print_r($e);
*/
}
echo "Trying to create account : ".$account_name."
";

try {
     $createAccountRes = $api->createAccount($account_name, "secret");
} catch (GuzzleHttp\Exception\ServerException $e) {
    echo "Some problem!
";
    echo "Account was NOT created!
";
    echo "Server thrown an Error, so may be this account already exist?
";
    echo "
";
//    echo print_r($e);
}

There are some things that make me unhappy...

Solution with exit(-1) seems be VERY ugly. But I don't know how to do it otherwise.
If server thrown an Exception for some other reason, not because account exists already...
I do not know, for example if the server is under maintenance mode.
I will not know this reason. I will still think that the problem is that the account exists.


Include meta in package derivations

I'm using the following to generate a basic BOM:
nix bundle .#my-package --bundler bundlers#toReport

The report will include node packages, but the they are missing license and maintainer information.  For example:
express-4.18.1 (no license) maintained by nobody

It would be nice if derivations generated for each package included meta extracted from their package.jsons.  Typically they will include license, author, and contributor information.  According to the package.json docs, licenses should generally use SPDX identifiers, which should hopefully make them easily to translate to meta.license.

I do this in nix-cargo-integration for Cargo projects. I can upstream that logic here and polish it up and add it to nodejs, rust and etc. subsystems.

@yusdacra that sounds like a lovely idea :)

I made #294 which adds licenses info for nodejs and rust. However, the report still doesn't work, because it doesn't handle multiple licenses (see https://github.com/NixOS/bundlers/blob/master/report/default.nix#L162-L168). But this isn't a dream2nix issue so it should be reported to bundlers repo.


Update angle and torsion distribution calculators

There are some scripts for calculating angle and torsion distributions in examples/evaluating_parameter_distributions. We probably want to generalize these and turn into functions for angle and torsion calculators - very useful if we are going to test out different torsion functional forms.

I can take a brief look at see how much revision they would take.

The new angle functions for the general heteropolymer update were merged in #38 (they are now located in analyze_foldamers).


Old tokens/pools showing in list for OG users

See also
https://github.com/HumbleOSS/humble-interface/issues/16

Added version what wipes out indexeddb storage if needed


Slowdown when calculating normals to draw

Hey Sebastian -
Great tool, thanks for sharing it!
When making a long path (many hundreds of units) the number of normals to calculate and draw was getting out of hand and slowing down the editor profoundly. Maybe consider not drawing each normal or showing every N where it's not doing too many?
for (int i = 0; i < normalsVertexPath.NumPoints; i++) {                         normalLines[i * 2] = normalsVertexPath.GetPoint (i);                         normalLines[i * 2 + 1] = normalsVertexPath.GetPoint (i) + normalsVertexPath.GetNormal (i) * globalDisplaySettings.normalsLength; }

Just wanted to follow up and say I realized after writing this that folding up the Normals foldout would make normals not draw, which is a workaround. However using the foldout as a display toggle is not obvious. Would suggest a checkbox here, possibly in the data object.


Implementation of AllSetTransformer

Initial commit of AllSetTransformer implementation for the ICML 2023 challenge.
We originally wanted to unify AllSet models with and without attention,  but given that their implementations greatly differ from each other we decided to add this other pull request for the transformer-based variant.

Hello @gbg141  ! Thank you for your submission. As we near the end of the challenge, I am collecting participant info for the purpose of selecting and announcing winners. Please email me (or have one member of your team email me) at<EMAIL_ADDRESS>so I can share access to the voting form. In your email, please include:

your first and last name (as well as any other team members)
the title of the model you implemented
the domain of the model you implemented
your pull request number (#128)

Before July 13, make sure that your submission respects all Submission Requirements laid out on the challenge page. Any submission that fails to meet this criteria will be automatically disqualified.

Hello @gbg141 !
Thank you for your submission! For the purposes of voting, could you please send me an email at<EMAIL_ADDRESS>with your (or your team members') full name and name of model implemented?
Thank you!


RFC: Catalog data cleaning

Description

This outline the rationale and a rough plan to perform a cleanup of the Catalog's database once and for all. Please leave your comments!
Reviewers

[ ] @AetherUnbound
[ ] @stacimc

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

I've been thinking over how best to implement this project for a long time, and here are some thoughts I had about it:
Constraints
When working with the upstream database, we have the following constraints:

There is a lot of data, more than 600 000 000 items in one table.
We have some indexes on the table. If we try to simply update all rows with, for example, a new license_url column, it will take a long time due to how Postgres works. When we run UPDATE on the table, Postgres creates a new row version and will need to update the index to point to that new row. There is a way of improving the UPDATE performance by using HOT updates, but that requires extra disk space and some expertise for tuning it (more about this in SO: https://stackoverflow.com/questions/71618615/does-postgresql-update-index-even-when-the-updated-columns-arent-the-indexed-on).
At the same time, selecting from the table might be slow due to point 1 and not enough indexes. This means that we cannot simply "select all the rows that have defective URLs and update them".

Prior work
Fortunately, all the data collected after the introduction of the ImageStore class is cleaned up.
The ImageStore class was created to move the API clean-up steps to the ingestion process in a cleaner way. After we've added the Audio, the clean-up code moved to MediaStore class.
The tags are cleaned up in the clean_media_metadata method:
https://github.com/WordPress/openverse-catalog/blob/f3799fc02c189848082decae60afaca7d9fefa2c/openverse_catalog/dags/common/storage/media.py#L92
Each media property has its own column type in the MediaStore. In the _create_tsv_row method, each value is cleaned using the appropriate column type's prepare_strings method. For instance, the URLColumn uses urls.validate_url_string method to add https, if possible, and to make the links absolute.
https://github.com/WordPress/openverse-catalog/blob/f3799fc02c189848082decae60afaca7d9fefa2c/openverse_catalog/dags/common/storage/columns.py#L492
There was a cleaner DAG created at CC that would select some rows in the upstream table, clean them up, and replace the upstream data with the cleaned data. This should work if we decide to use it, but I suspect it would take a long time due to the constraints.
Proposal
We can update the "cleaner DAG" to make the process of data cleanup faster by changing it in several ways:

we create a new database without the indexes - this will make writing to the database very fast.
we can use the backup parquet files to read from. This will make the reads fast. We would need to take into account the columnar structure of parquet vs the row-based structure of Postgres and any changes in the row from a TSV row.
we would need to update the add_item method for any new columns we decide to add (like the license_url), or old columns we decide to remove.
We would need to make sure that we don't miss any rows, save any rows that fail during the clean up step (for any reason), clearly log the time passed and expected time, can cancel the process at any time and hopefully resume where it stopped. Another consideration is how to make sure that the data that is collected during the run of this "cleaner DAG" is also saved to the new database without omissions or duplication.
Since this process can be expensive in terms of storage costs, it is important to decide what we want to change in the database table structure beforehand.

Note:
Due to the columnar structure of parquet file, I also had an idea of a possibly more efficient process of updating specific columns and not going through the data row by row. For example, if we could only look into the URL columns and run the URL cleanup on them. Or, read the 2 columns of license and license_version and write the 3rd column with the license_url. However, I learned that you cannot update the parquet files, only write new ones. And with 600 000 000 rows, it might be a very expensive TSV file to store.

@krysal could you add a due date to the PR description? my recommendation would be two more weeks, at most, but up to you. I just want to make sure reviewers have clear expectations on when they need to comment. Thanks!

@obulat Thank you for sharing your thoughts. Replying to some points:

Constraints

...
We have some indexes on the table. If we try to simply update all rows with, for example, a new license_url column, it will take a long time due to how Postgres works...

It should be noted that structural changes of DB tables are not included here (adding or modifying columns). I raised the flag for those on the Make Openverse Blog but I'm not planning to include migrations on this project.

Proposal

...
we can use the backup parquet files to read from. This will make the reads fast. We would need to take into account the columnar structure of parquet vs the row-based structure of Postgres and any changes in the row from a TSV row.

That is an interesting idea I haven't considered. Do you mean the inherited parquet files we have? I'm not aware of which part of the data those files cover but would be interesting to try how much it improves the performance of the cleaning process.

We would need to make sure that we don't miss any rows, save any rows that fail during the clean up step (for any reason), clearly log the time passed and expected time, can cancel the process at any time and hopefully resume where it stopped. Another consideration is how to make sure that the data that is collected during the run of this "cleaner DAG" is also saved to the new database without omissions or duplication.

Agree 💯

Note:
Due to the columnar structure of parquet file, I also had an idea of a possibly more efficient process of updating specific columns and not going through the data row by row. For example, if we could only look into the URL columns and run the URL cleanup on them.

This sounds like the most appealing part of the parquet files. I, admittedly, don't have much experience with columnar storage aside from theoretical knowledge, so we could create an issue to research this option and do some experiments.

However, I learned that you cannot update the parquet files, only write new ones. And with 600 000 000 rows, it might be a very expensive TSV file to store.

Presumably, it is only the data ingested prior to the inclusion of the MediaStore classes that needs cleaning, so it shouldn't be the whole dataset either.

@AetherUnbound

I would be curious to know if there was any information we could find on how long that workflow would take to run (and if it was ran at all).

We were informed that the DAG probably never ran, so I don't have information on much it would take (surely a considerable time). Regarding whether the ingestion and refreshing processes will need to stop, it depends on the strategy the cleaning DAG will use, if updates can be performed on the same table then I agree it may not need to be stopped, but @obulat notes suggest it will probably need to do a table switch, similar to that performed by the ingestion process, so in that case, we'd need to stop the DAGs in order not to lose info.

Since we already have the issues and the milestone, are we planning on having a technical implementation plan for this project as well?

This is the point where it becomes more noticeable that the heavy work of the project will concentrate on optimizing the cleaner DAG. I'd also like to be able to do it in batches, this sounds possible by making use of the created_on column. I can go into more details about the DAG in the coming days but estimating the duration of its execution can be pretty difficult without any running before.

@krysal could you add a due date to the PR description? my recommendation would be two more weeks, at most, but up to you. I just want to make sure reviewers have clear expectations on when they need to comment. Thanks!

I think we need at least 3 more weeks considering that the next two weeks will be AFK for many contributors.

Could we consider having the cleanup DAG take a provider name as configuration, and run cleanup for a single provider at a time as a way to minimize risk?

(Older) Flickr data often has problems with URLs, and selecting a single provider does not help when it has [checking the latest numbers] 468 372 078 (!) records.

It should be noted that structural changes of DB tables are not included here (adding or modifying columns). I raised the flag for those on the Make Openverse Blog but I'm not planning to include migrations on this project.

I understand the wish to minimize the amount of work in a single project to make it achievable. My reservation with this is that if we decide not to add or remove columns, we will have to repeat a very costly operation in the future.
With category column, it was easy to add it because we only added a blank column and did not back-fill. However, if we, for example, decide to add the license_url column in the future, and try to back-fill it using the data from license and license_version columns for all 600 million rows, it will take a really long time because all of the rows will need to be re-written due to the way Postgres handles UPDATE queries.
When running an UPDATE query, Postgres usually adds a new row with updated data and makes all of the indexes point to the new row in the table. This means a lot of work and extra space for every row we need to update.

That is an interesting idea I haven't considered. Do you mean the inherited parquet files we have? I'm not aware of which part of the data those files cover but would be interesting to try how much it improves the performance of the cleaning process.

Our upstream database is backed up to parquet files regularly, and I thought we could use the backup files for this...

Could we consider having the cleanup DAG take a provider name as configuration, and run cleanup for a single provider at a time as a way to minimize risk?

(Older) Flickr data often has problems with URLs, and selecting a single provider does not help when it has [checking the latest numbers] 468 372 078 (!) records.

100%, it definitely does not help speed up Flickr. I'm mainly suggesting this in response to concerns about testing -- maybe we can run the cleanup on a single (much, much smaller) provider first.

I think we need at least 3 more weeks considering that the next two weeks will be AFK for many contributors.

+1. I think we need that amount of time, given AFK and the amount of open questions about the alternatives @obulat has suggested using the backup parquet files 😮 It's a really interesting idea and I think the point about potentially needing to repeat a costly operation for license_url is a great one.
Regarding that approach: my assumption is that ingestion/data refresh would need to be halted during that process, which would be a significant con. Do you think that's the case @obulat or do you see a way around that?

I'm mainly suggesting this in response to concerns about testing -- maybe we can run the cleanup on a single (much, much smaller) provider first.

This seems like a great idea, to test on a tiny provider. It wouldn't be perfect but we could use that to make estimates about how long larger providers might take.
Something that could work similarly is to also have a max number of records to process, so we could try a config of like {provider: 'flickr', limit: 50} and see how long it takes, then make a projection about the entire Flickr dataset.

These are great ideas and important concerns raised here. Thank you all for the valuable comments 😄 It seems that this requires some investigation on how many rows need cleaning and more details on the approach the cleaner_image DAG would need. I'll think about those tasks and, in the meantime, draft this until collecting more information.