Add parameter loading + global properties component

As per the basic example, we want some way to load parameters from a config file and expose them as global properties.
Something along the lines of:
struct Parameters {
    random_seed: u64,
    population_size: usize,
    foi: f64,
    infection_period: u64
}
context.load_parameters<Parameters>("config.toml")

Which would then be available to any module.
Some global properties will need to be mutable.

For an initial implementation at this, I'd recommend the following:

Have people define a single Parameters struct for now
Use this to load / validate everything under aparameters property of given yaml config path. This should be some kind of trait extension to context like load_parameters
Give people immutable access to each property (don't allow changes / mutation events for now)

We will need to adjust the implementation to allow for mutation/events etc., but I think it shares a lot of the same requirements as person properties and we can update it later when we come up with what we want to do then


”Schitu” is listed in the drop-down menu

Description:
"Schitu" is listed in the drop-down menu alongside its corresponding county and commune, according to the law.
Precondition:
The website is up an running.

Step 1
Type "Schitu" in the search bar.

Expected results
The location is listed in the drop-down menu as "Comuna Schitu, GIURGIU".


[W5.11][W13-A1] Ang Jing Zhe

2 features: Adding sort command for addressbook & Add case-insensitivity to findCommand.
Updated minor compilation errors for previous LOs.

@AJZ1995
Some comments added. Please ack & update the PR after reading comments.

Ack


Refactor ODE Handler from Bintr to Handler

The ODE Handler should not be a bintr component in a fixed Pipeline/Branch location, but more of a pure batch-meta handler that can be added to the sink pads of Tiler's, Demuxer's and OSD components.
Need to refactor the ODE Handler and other components to support this.

@rjhowell44 not related to the ODE handler, but should this not also be the case for the MeterSinkBintr? This would allow one to attach it as a batch meta handler without tee-ing of and creating a seperate sink just to measure

@jlerasmus yes, no doubt... I'll rework this issue to include both. Thanks

@jlerasmus here's what I'm proposing,..
In hindsight, going with the term batch-meta-handler was a mistake as there are other things to do with the buffer than read/modify metadata...  object and frame capture are examples. Using pad-probe-handler is more accurate
So while I'm refactoring...
Implement new DSL PPH API with three derived classes... for now
// callback typedef for the custom pad-probe-handler
typedef boolean (*dsl_pph_custom_client_handler_cb)(void* buffer, void* client_data);

// callback typedef for the meter pad-probe-handler
typedef boolean (*dsl_pph_meter_client_handler_cb)
    (double* session_fps_averages, double* interval_fps_averages, uint source_count, void* client_data);

// new Custom PPH
DslReturnType dsl_pph_custom_new(const wchar_t name,
    dsl_pph_custom_client_handler_cb client_handler, void* client_data);

// new Meter PPH
DslReturnType dsl_pph_meter_new(const wchar_t* name, uint interval,
    dsl_pph_meter_client_handler_cb client_handler, void* client_data);

// new ODE PPH
DslReturnType dsl_pph_ode_new(const wchar_t* name);

All existing  *_batch_meta_handler_* services will be updated, for example
DslReturnType dsl_osd_batch_meta_handler_add(const wchar_t* name, uint pad,
    dsl_batch_meta_handler_cb handler, void* user_data);

DslReturnType dsl_osd_batch_meta_handler_remove(const wchar_t* name,
    uint pad, dsl_batch_meta_handler_cb handler);

will become
DslReturnType dsl_osd_pph_add(const wchar_t* name, const wchar_t* handler, uint pad);

DslReturnType dsl_osd_pph_remove(const wchar_t* name, const wchar_t* handler, uint pad);

All exisitng ode_handler and sink_meter APIs will be mapped over.

@rjhowell44 this makes sense.
My only remark is that client_data should probably be passed as parameter when adding the pad-probe-handler
DslReturnType dsl_osd_pph_add(const wchar_t* name, const wchar_t* handler, uint pad, void* client_data);

@jlerasmus I see where you're going with the above, however not all PPH's will require client callbacks and client data... the ODE PPH is one example.  I'm hoping most won't
For the case of the Custom, Meter, and others that do... if you wanted to use a single client_callback function with different client data for uniqueness, you would need to do something like the following.
retval = dsl_pph_custom_new('my-custom-pph-1', my_single_callback, my_custom_data_1)
retval = dsl_pph_custom_new('my-custom-pph-2', my_single_callback, my_custom_data_2)

hopefully that's not to convoluted?

@rjhowell44 you are right, this should/will only apply to specific PPH's.
For the custom-handler cases, and only on multiple usage might it feel convulated, so I think it is ok.

@jlerasmus this work is done, the three PPH types can be moved around from component to component and sink pad to source pad... and in doing so, a few issues/questions have come up which I will cover over on the ODE Open Items thread.
Closing this issue


create fibonacci

Want to create a fibonacci.

Would like to create Fibonacci

I Would like to create Fibonacci

I will be delighted to take this issue.I know to create it with c , python , javascript and c++ language


Unable to export Project file. unsupported operand type(s) for +: 'WindowsPath' and 'str'

Describe the bug
I tried to export Project file of my partially reviewed project. It failed and displays the following message:

unsupported operand type(s) for +: 'WindowsPath' and 'str' Please try again.

However, it successfully export my datasets without any issue. Including both full and relevant datasets in all formats.
To Reproduce

Go to Projects
Click "Export" on desired project
Select "Project" file
Click "Export"

Screenshots
n.a.
Version information

OS: Microsoft Windows 10 Home Single Language
Browser: Mozilla Firefox
ASReview version: 1.6.3

Additional context
Here's the output from cmd
ERROR:asreview.webapp.app:Exception on /api/projects/4820717b801b45a0bfdfe96439c7ae87/export_project [GET]
Traceback (most recent call last):
  File "C:\Users\USER\.pyenv\pyenv-win\versions\3.11.0b4\Lib\site-packages\flask\app.py", line 1473, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\.pyenv\pyenv-win\versions\3.11.0b4\Lib\site-packages\flask\app.py", line 882, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\.pyenv\pyenv-win\versions\3.11.0b4\Lib\site-packages\flask_cors\extension.py", line 194, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
                                                ^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\.pyenv\pyenv-win\versions\3.11.0b4\Lib\site-packages\flask\app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\.pyenv\pyenv-win\versions\3.11.0b4\Lib\site-packages\flask\app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\.pyenv\pyenv-win\versions\3.11.0b4\Lib\site-packages\flask_login\utils.py", line 290, in decorated_view
    return current_app.ensure_sync(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\.pyenv\pyenv-win\versions\3.11.0b4\Lib\site-packages\asreview\webapp\authentication\decorators.py", line 40, in decorated_function
    return f(project, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\.pyenv\pyenv-win\versions\3.11.0b4\Lib\site-packages\asreview\webapp\api\projects.py", line 1253, in export_project
    project.export(tmpfile)
  File "C:\Users\USER\.pyenv\pyenv-win\versions\3.11.0b4\Lib\site-packages\asreview\project.py", line 653, in export
    shutil.make_archive(export_fp_tmp, "zip", root_dir=export_fp_tmp)
  File "C:\Users\USER\.pyenv\pyenv-win\versions\3.11.0b4\Lib\shutil.py", line 1130, in make_archive
    filename = func(base_name, base_dir, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\.pyenv\pyenv-win\versions\3.11.0b4\Lib\shutil.py", line 977, in _make_zipfile
    zip_filename = base_name + ".zip"
                   ~~~~~~~~~~^~~~~~~~
TypeError: unsupported operand type(s) for +: 'WindowsPath' and 'str'
ERROR:root:unsupported operand type(s) for +: 'WindowsPath' and 'str'

Thanks for the detailed bug report. I haven't seen this before, strangely.
I think shutil.make_archive(export_fp_tmp, "zip", root_dir=export_fp_tmp) should be able to handle Path objects?
Can a Windows user with 1.6.3 reproduce this issue? @Rensvandeschoot @PeterLombaers @VeenDuco

I tried to reproduce, but for me the export worked fine.
Versions

Windows 10
Python: 3.11
ASReview 1.6.3
Browser: Chrome

@Grapes178 Your Python version most likely causes the issue. You are using a release candidate with a bug. Please use ASReview with a stable Python version, preferably the latest update.
See https://github.com/python/cpython/issues/94844 for more details about the bug.


FRQ: add /submits command

which would do what /knobs does, but for submit keywords, which we should be able to fish out of
https://htcondor.readthedocs.io/en/latest/man-pages/condor_submit.html

The second part of the commit message for 7391a06 should have been for #22.

So the thread window is narrow enough that even if we adjusted the hard-wrap to usually fit, the second and subsequent answers would be hard to tread.  We could print the first entry and if there are more, a link to the web page; we could print the first entry with a button that sends the rest as DMs; or we could give the bot a channel to be verbose in, and a print a link to that if there are multiple entries.  Another option might be to publicly reply with the first entry and DM the original requester the remainder, although that might devolve back into the second option pretty quickly.
For multiple replies, I guess we could also just ask the original requester which one to dump back to the channel.  (Truncate the individual replies to four lines or so, print them all with a button that says 'this one!' that just duplicates the (hidden) whole reply into the channel.)  This wouldn't really for long replies, though.

I think a link makes the most sense. The important thing is not really that the information literally shows up in Slack (although that's nice when it happens), but that we "find" the information for the requester. The first few lines followed by a direct link to the right submit option is good enough for me.

OK, I'm declaring victory.


Using document set function, got Error: 5 NOT_FOUND

What you're trying to do
I was trying to integrate firebase into Nuxt3 server api and following the instruction from official document. Then I was at the step of Add Data, but I got Error: 5 NOT_FOUND. After searched relative information, still couldn't solve this problem.

What code you've already tried
Environment:

OS: macOS Sonoma 14.3.1
Node Version: v20.11.0
NPM Version: v10.2.4
Package.json: "firebase-admin": "^12.0.0", "nuxt": "^3.10.3", "vue": "^3.4.21", "vue-router": "^4.3.0"

Code:

import path from 'path';
import fs from 'fs';
import { initializeApp, applicationDefault, cert } from 'firebase-admin/app';
import { getFirestore, Timestamp, FieldValue, Filter } from 'firebase-admin/firestore';

export default defineEventHandler(async (event) => {
    try {
        const config = useRuntimeConfig(event);
        const filePath = path.resolve(process.cwd(), config.firebaseKey)
        const serviceAccount = JSON.parse(fs.readFileSync(filePath, 'utf-8'))
        initializeApp({ credential: cert(serviceAccount) })

        const db = getFirestore();
        const docRef = db.collection('/users').doc('alovelace');

        await docRef.set({
        first: 'Ada',
        last: 'Lovelace',
        born: 1815
        });
        return 'hello';
    } catch (error) {
        throw error;
    }
});

Any error messages you're getting

error:  Error: 5 NOT_FOUND:
    at callErrorFromStatus (/Users/hank/help-u-focus/node_modules/@grpc/grpc-js/src/call.ts:82:17)
    at Object.onReceiveStatus (/Users/hank/help-u-focus/node_modules/@grpc/grpc-js/src/client.ts:360:55)
    at Object.onReceiveStatus (/Users/hank/help-u-focus/node_modules/@grpc/grpc-js/src/client-interceptors.ts:458:34)
    at Object.onReceiveStatus (/Users/hank/help-u-focus/node_modules/@grpc/grpc-js/src/client-interceptors.ts:419:48)
    at <anonymous> (/Users/hank/help-u-focus/node_modules/@grpc/grpc-js/src/resolving-call.ts:132:24)
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
for call at
    at ServiceClientImpl.makeUnaryRequest (/Users/hank/help-u-focus/node_modules/@grpc/grpc-js/src/client.ts:325:42)
    at ServiceClientImpl.<anonymous> (/Users/hank/help-u-focus/node_modules/@grpc/grpc-js/src/make-client.ts:189:15)
    at /Users/hank/help-u-focus/node_modules/@google-cloud/firestore/build/src/v1/firestore_client.js:239:29
    at /Users/hank/help-u-focus/node_modules/google-gax/build/src/normalCalls/timeout.js:44:16
    at repeat (/Users/hank/help-u-focus/node_modules/google-gax/build/src/normalCalls/retries.js:80:25)
    at /Users/hank/help-u-focus/node_modules/google-gax/build/src/normalCalls/retries.js:119:13
    at OngoingCallPromise.call (/Users/hank/help-u-focus/node_modules/google-gax/build/src/call.js:67:27)
    at NormalApiCaller.call (/Users/hank/help-u-focus/node_modules/google-gax/build/src/normalCalls/normalApiCaller.js:34:19)
    at /Users/hank/help-u-focus/node_modules/google-gax/build/src/createApiCall.js:108:30
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Caused by: Error
    at WriteBatch.commit (/Users/hank/help-u-focus/node_modules/@google-cloud/firestore/build/src/write-batch.js:433:23)
    at DocumentReference.set (/Users/hank/help-u-focus/node_modules/@google-cloud/firestore/build/src/reference.js:403:27)
    at Object.handler (/Users/hank/help-u-focus/server/api/hello.get.js:16:1)
    at file:///Users/hank/help-u-focus/node_modules/h3/dist/index.mjs:1890:43
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.handler (file:///Users/hank/help-u-focus/node_modules/h3/dist/index.mjs:1962:19)
    at async Server.toNodeHandle (file:///Users/hank/help-u-focus/node_modules/h3/dist/index.mjs:2249:7) {
  code: 5,
  details: '',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} },
  note: 'Exception occurred in retry method that was not classified as transient'
}

It's working now. I'm not sure which step resulted in the creation of a new second project in Google Cloud. However, when I used the service account key from this new second project, everything started functioning properly.
Yesterday, I was troubleshooting an error prompted by the firebase-admin module. Following the guidance, I navigated to the Google Cloud platform, created a new project, and assigned a service account key. But it didn't work.
Today, when I navigated to the Firebase console > project settings > service accounts tab, I suddenly noticed there was a new second project. Eventually, using the service account key from this new project resolved the issue.


Fixes for NREs and ArgumentExceptions

Working back through these now, the null checks and stuffs Ive cleaned up in a bigger sweep. Thanks a heap for the input here I used this and two other PRs to get all the edges
The button toggle one I didnt take as I cant seem to trigger that Argu ent exception and the PR doesnt have any style bits in it. Ill keep an eye and might trigger something more in a bit


How to share with both URL and message?

I read the document. It says :

NOTE: If both message and url are provided, url will be concatenated to the end of message to form the body of the message. If only one is provided it will be used

But when I try to share a video with a URL and a message. But It shows error:

Error: "Failure from system"

This is my code:
const URI64 = `data:video/mp4;base64,${URI}`;   Share.open({message: "Paris"+ URI64,title : "Share a video"})
URI is the base64 from local uri
Does someone can help me about that?

Hello @EnclaveAnthony I guess that documentations are a bit misleading. Did you try to share only message or only uri?
Could you please test that for us and bring the results? Thanks.

Hi @jgcmarins.
When I try to share only message or only uri. It works perfectly.

Hi @jgcmarins.
This is my code :
const URI = base64.encode(this.state.uri)         const URI64 = `data:video/mp4;base64,${URI}`         Share.open({url : URI64, message: "Some message",title : "Title"})
It only works well on a few apps like Mail, Imessage.
On Facebook, Instagram, and Whatsapp, It only has video and doesn't has the message.

Update : Share only message not work on facebook.

Same problem on iOS 13.

I read the document. It says :

NOTE: If both message and url are provided, url will be concatenated to the end of message to form the body of the message. If only one is provided it will be used

But when I try to share a video with a URL and a message. But It shows error:

Error: "Failure from system"

This is my code:
const URI64 = `data:video/mp4;base64,${URI}`; Share.open({message: "Paris"+ URI64,title : "Share a video"})
URI is the base64 from local uri
Does someone can help me about that?

Myself also having the same issue - "Failure from system" what did you done to fix this issue
here is my code :
   RNFS.readFile(url, 'base64')
            .then(base64 => {
                // console.warn(base64);

                let fileType = 'video/mp4'
                let base64Data = 'data:${fileType};base64,' + base64; // this works as url paramet  er
                let fileUrl = "file://" + url; // using this as url does not work

                let shareOptions = {
                    // title: 'Share',
                    message: base64Data,
                    // type: fileType,
                    // url: base64Data,

                }
                Share.open(shareOptions).then(res => {

                });

            })


problem of indicate selection at vscode.window.showQuickPick() with octicons

VSCode Version: 1.27.2(lang:ja), 1.28(insider, lang:en)
OS Version: Windows, Mac

Steps to Reproduce:

run this code in extension

        await vscode.window.showQuickPick
        (
            [
                {
                    label: `$(primitive-dot) $(file-text) Untitled-1`,
                    description: "description",
                    detail: "detail"
                },
                {
                    label: `$(primitive-dot) $(file-text) Untitled-2`,
                    description: "description",
                    detail: "detail"
                }
            ],
            {
                placeHolder: "Select a unsaved file",
            }
        );

type unt

Result:
The display of octicon will be broken and the wrong place will be selected and displayed.

Does this issue occur when all extensions are disabled?: No( this problem is at api for extensions )

@chrmarti I am quite sure I fixed this a while back for normal quick open, maybe it is a regression from the new quick input widget?

My change was: https://github.com/Microsoft/vscode/commit/ee9f1e7e4179ad68266150bfb272bc37a403982b


RTSP Link include &param= parametr

Hello!
I have rtsp link includes parametr '&subtype=0', full link as: rtsp://videouser:password@<IP_ADDRESS>:31003/cam/realmonitor?channel=1&subtype=0
If i'm add this link over [PUT] /api/streams?src=rtsp://videouser:password@<IP_ADDRESS>:31003/cam/realmonitor?channel=1&subtype=0 - programm trimm '&subtype=0' and save only rtsp://videouser:password@<IP_ADDRESS>:31003/cam/realmonitor?channel=1

You should use some URL query encoding function.


Write only INFO file?

Is it possible to write only the INFO file and suppress others (WARNING, ERROR)? INFO will contain the information from everything, so I don't want to have the other files. Does GLOG do this automatically?

One could use a custom LogSink to redirect the output into a single file. See code documentation for details.


JavaScript

JavaScript
User Story
As a boot camp student
I want the prework notes to be structured on a webpage
So that I can easily find and read the information
Acceptance Criteria
GIVEN a Prework Study Guide website
WHEN I view the study guide
THEN I can see the four topics I learned listed along with a suggestion on what I should study first

Completion of this step with adding in Javascript code


[l10n] Improve Russian translation

[x] I have followed (at least) the PR section of the contributing guide.

This is a quick fix for Russian text lines in library localization.
However, there is an issue with declination. In Russian you can't simply add an ending to noun to make it plural (as in English). There are two different endings for plural nouns (and sometimes word alters a bit).
Example:
2-4 stars translates as 2-4 звезды, but 5, 6, 7... 20 stars will be звёзд. Then again 21 stars translates as 21 звезда.
Common solution here is to define function that takes three word forms and a number and returns suitable one.
Is it ok, if I put such function into /utils subfolder?
Closes #18421

@gmltA Regarding the plural handling, could you update the getLabelText function to account for all the cases? I think that we should defer the creation to helpers to later (when we see the pain point growing)

@oliviertassinari roger that!

About plural form it is a nice case. Our modules are separate esm bundles? If so we can use packages like this https://www.npmjs.com/package/plural-ru which will automatically resolve the plural form

@dmtrKovalenko I had a thought about utilizing some third-party for this, but as @oliviertassinari mentioned, for now it is not something vital (literally one case in the whole repository).

Awesome, thank you guys.


Tx STX amount + fee higher than User's account Balance

Today/yesterday discovered a case where the the user does not have sufficient balance but still was able to proceed with transaction.

Tx nonce 46 is for 2513 stx + 0.6144 fee, however the user only has balance of 2513.02 stx in balance, which is causing the wallet to be clogged up.

In Theory, the wallet shouldn't let the user do that in the first place, whether this is the first attempt or the increase fee attempt. (from my experience most likely the user had smaller fee, sufficient fund, but pending too long, so the user attempts to increase fee to an amount over their balance unknowingly, because all they do is press increase fee, pick a random X5 or X 10 amount press confirm. At this point the wallet in theory should not allow the user to confirm, but it seems like it does currently)

why does this tx not fail, it just keeps pending been for 1 + day now, shouldn't tx realize the user does not have sufficient fund and fails taking the fee off balance?

Thanks Team.

I cannot reproduce this. The validation on the form catches this...

Nonce 46 tx and 47 dropped off, creating nonce gap now..
Re the original subject, user said "I saw it was taking too long, maybe around 1 hour afterwards"
Full convo in Discord: https://discord.com/channels/621759717756370964/625538774230892545/966792480077000744


Definitions in

Vue version
3.2.37
Link to minimal reproduction
https://sfc.vuejs.org/#eNp9kE1ugzAQha8y9YZECljdUpKqu96gG28oDIHIfxobWgn57h0giaJW6sb2m+f59GZm8eZ9MY0oSlGFhgYfIWAc/UnZwXhHEWYg7CBBR85Axl8zZZVtnA0RTDjDcfF32Ttq7eDDkW6fsr2yldxwDGIR0XhdR2QFUPXPp3lem1OqJKu1Olg/Rphy41rURyXYV4KtSj50s7xzAfB7jdhiV4+aoy41AONGG7Hd7W8FgNgPodjSbklfs5fNS8vFxz2vOIht8tzUvrgEZ3k3K0hdjaBEeUMrwRtZtBJ9jD6UUoauWTZ6CYWjs+RXQZxnMFhgMPknua+AxGAlDg8MycUJKSe0LRLSf8xfX/9wryMlkX4Ach+j6Q==
Steps to reproduce
Use <script setup> and options API both, and the production build will ignore whatever is defined in <script setup>.
What is expected?
If the dev build supports both APIs working simultaneously without any warning, the production builds should also support it.
What is actually happening?
Options API in production build doesn't recognize whatever is defined in <script setup>.
System Info
No response
Any additional comments?
One can switch the mode in the provided SFC playground and see the difference.

Probably related to #6242

or more closely, #6241
None of these use case were intended for <script setup> so there's likely a couple of variations of this same basic problem to be found.

Any update on this? We have a huge project that utilizes both Options API and Composition API. If a solution is likely not to be available in the foreseeable future, then maybe anyone here could help with how to disable dev-tools from the dev build?

We have a huge project that utilizes both Options API and Composition API.

Are you saying that you have build lots of components with the above pattern and haven't tested a prod build until now?

If a solution is likely not to be available in the foreseeable future

It might take a little while till work on this, as it would likely require a change in the build tooling, possibly. We also need to discuss how we want to approach this issue generally as I need to stress again, this use case was never intended.

maybe anyone here could help with how to disable dev-tools from the dev build?

There's no way to do that, really.
You can make the compiler compile more like in dev if you are using Vite, though it's a hack and I can't say I recommend it:
// https://vitejs.dev/config/
export default defineConfig({
  plugins: [
    vue({
      // @ts-ignore
      devServer:
        process.env.NODE_ENV === 'production'
          ? {
              config: {
                base: '/',
                server: {
                  hmr: false,
                },
              },
            }
          : undefined,
    }),
  ],
  // ....

When there's a devServer option, Vite will not inline the render function for production. The additional options are there because if that option exists, the compiler will read these two properties during compilation.
Warning This is a hack, I haven't tested it thoroughly. Do with this what you think is best for your situation.

Are you saying that you have build lots of components with the above pattern and haven't tested a prod build until now?

Yes, it's an incoming project, so we didn't have control over it before the previous dev handed it over to us. So when we ran the production build, it failed.
Thanks for the snippet. Unfortunately, the project uses Laravel Mix.

Sorry to bug you guys.
Any news on this?

No. As I'm worried you are waiting for a solution that will never arrive, let me specify how we see this situation:

the dev behavior is unintended and needs to be fixed.
the prod behavior is correct

So in case you are waiting for a fix that will make your above code work, I'm sorry to disappoint you. You will need to fix the code on your end. If you need support on how to approach this, feel free to ask in a discussion thread or on discord.

I am not waiting for the above code to work in production, as we have already moved the existing codebase to composition API.
I am waiting for a fix for potential app-breaking differences between the dev and production build.
If this behavior is unintended, then either documentation should cover it:
https://vuejs.org/api/sfc-script-setup.html
Or the dev build should break, just like the production build.

I have the same issue, when I updated Vite to v3.3.2 and @vue/compiler-sfc to v3.2.41, it show error like this:
Error: [@vue/compiler-sfc] <script setup> cannot contain ES module exports. If you are using a previous version of <script setup>, please consult the updated RFC at https://github.com/vuejs/rfcs/pull/227.


Add options for OOBM action commands

Description
Currently the UI allows to call OOBM actions, however, the ADMIN must fill a texbox with the command which might lead to issues in case the letters are not all capital or the command does not exist.
To help ADMINs, this PR adds a options list with the supported Action commands.
Screenshot

How thios has been tested
Execute OOBM actions via the UI.

@blueorangutan package

@GabrielBrascher a Jenkins job has been kicked to build primate packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark:centos :heavy_check_mark:debian :heavy_check_mark:archive.
QA: http://primate-qa.cloudstack.cloud:8080/client/pr/894 (JID-3735)

If old UI gave options, we can treat it as bug - let's merge - LGTM.


Add support for Camel-NATS

I'll provide a PR :-)

Reopened. We need to figure out what's going on on a real server, before we can take this in.

@tdiesler, sure.
We need to allow the CI server to connect to the demo.nats.io server.

Even with a local embedded server we have some issues in the Jenkins environment. Here's the gnatsd log:
[14109] 2016/05/12 05:22:28.437577 [INF] Starting nats-server version 0.8.0
[14109] 2016/05/12 05:22:28.437633 [DBG] Go build version go1.6.2
[14109] 2016/05/12 05:22:28.437640 [INF] Listening for client connections on <IP_ADDRESS>:4222
[14109] 2016/05/12 05:22:28.437719 [DBG] Server id is OsPblhsIOBK6bWyYQIthi8
[14109] 2016/05/12 05:22:28.437724 [INF] Server is ready
[14109] 2016/05/12 05:22:29.226172 [DBG] <IP_ADDRESS>:38103 - cid:1 - Client connection created
[14109] 2016/05/12 05:22:29.250122 [DBG] <IP_ADDRESS>:38104 - cid:2 - Client connection created
[14109] 2016/05/12 05:22:29.257047 [TRC] <IP_ADDRESS>:38103 - cid:1 - ->> [CONNECT {"verbose":false,"pedantic":false}]
[14109] 2016/05/12 05:22:29.262636 [TRC] <IP_ADDRESS>:38104 - cid:2 - ->> [CONNECT {"verbose":false,"pedantic":false}]
[14109] 2016/05/12 05:22:29.262673 [TRC] <IP_ADDRESS>:38104 - cid:2 - ->> [PUB test 7]
[14109] 2016/05/12 05:22:29.262685 [TRC] <IP_ADDRESS>:38104 - cid:2 - ->> MSG_PAYLOAD: [message]
[14109] 2016/05/12 05:22:29.262899 [TRC] <IP_ADDRESS>:38103 - cid:1 - ->> [SUB test   1]
[14109] 2016/05/12 05:22:33.229542 [TRC] <IP_ADDRESS>:38103 - cid:1 - ->> [PING]
[14109] 2016/05/12 05:22:33.229580 [TRC] <IP_ADDRESS>:38103 - cid:1 - <<- [PONG]
[14109] 2016/05/12 05:22:33.230181 [TRC] <IP_ADDRESS>:38104 - cid:2 - ->> [PING]
[14109] 2016/05/12 05:22:33.230192 [TRC] <IP_ADDRESS>:38104 - cid:2 - <<- [PONG]
[14109] 2016/05/12 05:22:37.229627 [TRC] <IP_ADDRESS>:38103 - cid:1 - ->> [PING]
[14109] 2016/05/12 05:22:37.229677 [TRC] <IP_ADDRESS>:38103 - cid:1 - <<- [PONG]
[14109] 2016/05/12 05:22:37.229738 [TRC] <IP_ADDRESS>:38104 - cid:2 - ->> [PING]
[14109] 2016/05/12 05:22:37.229745 [TRC] <IP_ADDRESS>:38104 - cid:2 - <<- [PONG]
[14109] 2016/05/12 05:22:39.267179 [TRC] <IP_ADDRESS>:38103 - cid:1 - ->> [PING]
[14109] 2016/05/12 05:22:39.267207 [TRC] <IP_ADDRESS>:38103 - cid:1 - <<- [PONG]
[14109] 2016/05/12 05:22:39.269081 [TRC] <IP_ADDRESS>:38103 - cid:1 - ->> [UNSUB 1 0]
[14109] 2016/05/12 05:22:39.269096 [TRC] <IP_ADDRESS>:38103 - cid:1 - <-> [DELSUB 1]
[14109] 2016/05/12 05:22:39.269112 [TRC] <IP_ADDRESS>:38103 - cid:1 - ->> [PING]
[14109] 2016/05/12 05:22:39.269117 [TRC] <IP_ADDRESS>:38103 - cid:1 - <<- [PONG]
[14109] 2016/05/12 05:22:39.271324 [DBG] <IP_ADDRESS>:38103 - cid:1 - Client connection closed

The message ends up being sent before the consumer has subscribed to the topic. I don't see a non-hacky way of the integration test ensuring the consumer has subscribed to the topic before sending messages.
This is something that should be done at the camel component level IMO.

I'm looking in to the camel-nats code. I will add a field for the sid that will be set when the the consumer successful subscribed to the topic.
Anyway this will be something that will be available in 2.17.2

The component updates seems to have worked. I tested 2.17.x and 2.18.x through Jenkins and it worked fine.
Therefore, I rebased the original work for this (with some mods) to the wfc 2.17.x. and 2.18.x branches.
Leaving this issue open as a reminder to configure the Jenkins jobs to start up gnatsd prior to builds running.

Great :-)


Geocoding download for bbox

in SDK side: packagemanager should enable to download offline geocode packages for BBOX areas (e.g. cities).
in server side: there is special extract API script which takes copy of corresponding country file (with restriction that bbox must be in single country) and deletes all data outside defined bbox.

Customer case: Lonely Planet

Part of SDK 4.1.0RC1


WebAssembly smart contract and metadata documentation

Hello! I am working on getting a proof-of-concept (ERC20) Substrate pallet smart contract up and running using Grain (which uses WebAssembly as its sole compiler target). I understand that ink! is intended to be the primary way to write Substrate smart contracts, but, as one of the goals of these WebAssembly-based runtimes is to be able to directly leverage software written in Rust/C/C++/Grain/AssemblyScript/others, I think it would be helpful if developers could better understand the WASM ABI/Contract Metadata API that Substrate intends to use.
As for my proof-of-concept, I have gotten most of the way there, but there are some documentation gaps which are preventing me from getting it over the finish line. While I am looking for the answers to these questions, they are also meant to illustrate the types of hurdles that outside users face when trying to develop for the Substrate pallet framework.
First, here are some useful things which I have learned from the Substrate source code but I could not find documentation for (which I hope are not incorrect):

Each WASM module should export exactly two functions: call and deploy. Both of these take no arguments and return i32 values (Question: What is the significance of these values?). When the contract is initially created on the chain, deploy is called, and can do any initial setup (i.e. this is essentially the "constructor" from ink!). When a contract is sent a message, the call function is invoked. (Question: How does call determine which message it's been sent? I presume this has something to do with the selector in the contract metadata, but what are the precise mechanics of this?)
The list of environment-provided WASM functions can be found in this file. They all come from the following modules: seal0, seal1, __unstable__. If a sealN function is deprecated, then its seal(N+1) variant should be used instead.
seal_set_storage and friends use keys which are 32 bytes long.
The WASM file can use a start section to indicate a function which should be executed at startup.
The references to types inside of .contract files are one-indexed.

There are still some open questions which I have not been able to find an answer to, though:

Is the start section executed for each deploy and each call (the alternative being that the linear memory of the WASM module is snapshotted and restored)? I assume that the answer is yes, but I thought I'd double-check.
What should go inside of the .contract file? I imagine that ink!-generated ones are not the optimal minimal example for this, but I do not see any other sources. For example:

The source section has a hash value. What is this a hash of, and which hash function does it use?
To which schema should the entries in type conform? I some types have a path value (e.g. "path": ["ink_env", "types", "AccountId"]); what does this mean? Is it required?
What is the purpose of the storage section? Is it required? For example, my ERC20 contract simply uses the seal_set_storage and seal_get_storage functions to store balances (with the addresses as keys). How would this be represented in the storage section? What is the layout specifying? Why does the ink!-generated ERC20 .contract file have such complex storage layouts for its maps?
Why does ink! use complex types (variants, etc) in its storage values, rather than just primitive types? For example, it uses this for the balance table:

{
  "layout": {
    "struct": {
      "fields": [
        {
          "layout": {
            "struct": {
              "fields": [
                {
                  "layout": {
                    "cell": {
                      "key": "0x0100000000000000000000000000000000000000000000000000000000000000",
                      "ty": 2
                    }
                  },
                  "name": "header"
                },
                {
                  "layout": {
                    "struct": {
                      "fields": [
                        {
                          "layout": {
                            "cell": {
                              "key": "0x0200000000000000000000000000000000000000000000000000000000000000",
                              "ty": 3
                            }
                          },
                          "name": "len"
                        },
                        {
                          "layout": {
                            "array": {
                              "cellsPerElem": 1,
                              "layout": {
                                "cell": {
                                  "key": "0x0200000001000000000000000000000000000000000000000000000000000000",
                                  "ty": 4
                                }
                              },
                              "len":<PHONE_NUMBER>,
                              "offset": "0x0300000000000000000000000000000000000000000000000000000000000000"
                            }
                          },
                          "name": "elems"
                        }
                      ]
                    }
                  },
                  "name": "entries"
                }
              ]
            }
          },
          "name": "keys"
        },
        {
          "layout": {
            "hash": {
              "layout": {
                "cell": {
                  "key": "0x0300000001000000000000000000000000000000000000000000000000000000",
                  "ty": 9
                }
              },
              "offset": "0x0200000001000000000000000000000000000000000000000000000000000000",
              "strategy": {
                "hasher": "Blake2x256",
                "postfix": "",
                "prefix": "0x696e6b20686173686d6170"
              }
            }
          },
          "name": "values"
        }
      ]
    }
  },
  "name": "balances"
}

☝ This leaves me with a few questions:

Why does this need to be declared at all (see previous question)? Is this not an implementation detail?
Am I correct in assuming that this is some sort of hashmap spec (like, the address gets hashed and these structs provide a level of indirection to avoid any problems with hash collisions)?

What goes inside of the spec? At a high level, I understand the answer to this (it contains the constructor and any callable functions), but what is the schema?
How does the selector for each message type manifest in the WASM module? I am guessing that seal_input contains the selector as its first entry, but is that the case?

The seal_XX functions use a pattern of writing their results to a provided output buffer but often do not have a way to prompt them for the amount of data which will be written. What size scratch buffer should be used to ensure that these operations will not overrun?
How many bytes are values such as addresses, balances, etc.? I have gleaned that this appears to be configurable at the Substrate level, but, if so, ink! makes some decision on these sizes by default. How many bytes should we expect these values to be?

Again, I understand that ink! is the intended language for writing Substrate smart contracts, but it would be a huge boon to the Substrate user community if these types of documentation gaps were filled so that others could target the platform. Would you consider putting something together?
Appendix
For context, here is the current draft of the Grain program I have put together, which loosely illustrates how I would expect to be able to interact with the platform (not shown: the API bindings I wrote around the seal_XX functions):
/* grainc-flags --no-gc */
import Substrate from "../lib/substrate/substrate"
import {KeyNotFound} from "../lib/substrate/substrate"
import Bytes from "bytes"
import Int32 from "int32"
import Utils from "./utils"

// [TODO] When grain supports native uint256 values, switch
let _ERC20_SUPPLY_KEY = Bytes.make(32)
Bytes.fill(3l, _ERC20_SUPPLY_KEY)

enum Action {
  TotalSupply,
  BalanceOf,
  Transfer,
  TransferFrom,
  Approve,
  Allowance,
}

// Grain doesn't support automatic generation between ints and enums, so we need a conversion function:
let toAction = (n: Int32) => {
  if (n == 0l) {
    TotalSupply
  } else if (n == 1l) {
    BalanceOf
  } else if (n == 2l) {
    Transfer
  } else if (n == 3l) {
    TransferFrom
  } else if (n == 4l) {
    Approve
  } else if (n == 5l) {
    Allowance
  } else {
    Substrate.debugMessage("Bad code: " ++ toString(n))
    fail("Bad code: " ++ toString(n))
  }
}

let getBalanceOrZero = (address) => {
  match(Substrate.getStorage(address)) {
    Ok(d) => d,
    Err(KeyNotFound) => {
      let ret = Bytes.make(16) // balance bytes = 16
      Bytes.fill(0l, ret)
      ret
    },
    Err(e) => fail("Error")
  }
}

let mergeToSha256 = (a, b) => {
  let tmp = Bytes.make(Bytes.length(a) + Bytes.length(b))
  Bytes.move(0, 0, Bytes.length(a), a, tmp)
  Bytes.move(0, Bytes.length(a), Bytes.length(b), b, tmp)
  Substrate.hashSha2256(tmp)
}

// dispatch helper function for call()
let handle = (action: Action, input) => {
  match(action) {
    TotalSupply => {
      match(Substrate.getStorage(_ERC20_SUPPLY_KEY)) {
        Ok(d) => d,
        Err(e) => {
          fail("Error")
        }
      }
    },
    BalanceOf => {
      let owner = Bytes.slice(1, 33, input)
      getBalanceOrZero(owner)
    },
    Transfer => {
      let to = Bytes.slice(1, 33, input)
      let value = Bytes.slice(33, 49, input)
      let caller = Substrate.caller()
      let balanceFrom = getBalanceOrZero(caller)
      let balanceTo = getBalanceOrZero(to)
      let (>=) = Utils.int128Gte
      let (+) = Utils.int128Add
      let (-) = Utils.int128Sub
      if (balanceFrom >= value) {
        Substrate.setStorage(caller, balanceFrom - value)
        Substrate.setStorage(to, balanceTo + value)
      }
      Bytes.empty
    },
    TransferFrom => {
      let owner = Bytes.slice(1, 33, input)
      let to = Bytes.slice(33, 65, input)
      let value = Bytes.slice(65, 81, input)
      let caller = Substrate.caller()
      let balanceOwner = getBalanceOrZero(owner)
      let balanceTo = getBalanceOrZero(to)
      let allowanceKey = mergeToSha256(owner, caller)
      let allowance = getBalanceOrZero(allowanceKey)
      let (>=) = Utils.int128Gte
      let (+) = Utils.int128Add
      let (-) = Utils.int128Sub
      if (balanceOwner >= value && allowance >= value) {
        Substrate.setStorage(owner, balanceOwner - value)
        Substrate.setStorage(to, balanceTo + value)
        Substrate.setStorage(allowanceKey, allowance - value)
      }
      Bytes.empty
    },
    Approve => {
      let spender = Bytes.slice(1, 33, input)
      let amount = Bytes.slice(33, 49, input)
      let caller = Substrate.caller()
      let allowanceKey = mergeToSha256(caller, spender)
      Substrate.setStorage(allowanceKey, amount)
      Bytes.empty
    },
    Allowance => {
      let owner = Bytes.slice(1, 33, input)
      let spender = Bytes.slice(33, 65, input)
      let allowanceKey = mergeToSha256(owner, spender)
      getBalanceOrZero(allowanceKey)
    },
   }
}

export let call: (() -> WasmI32) = () => {
  let input = Substrate.input()
  let action = toAction(Bytes.getInt8U(0, input))
  let output = handle(action, input)
  Substrate.return([], output) // <- does not return
  0n
}

export let deploy: (() -> WasmI32) = () => {
  // Initialize total supply
  let totalSupply = Substrate.valueTransferred()
  Substrate.setStorage(_ERC20_SUPPLY_KEY, totalSupply)
  // Fill wallet of contract caller
  Substrate.setStorage(Substrate.caller(), totalSupply)
  0n
}

CC @athei

Hi @peblair thanks for taking the time to give us this very useful feedback. Let me start off with saying that supporting a diverse set of programming languages was and is an important design goal of pallet_contracts and we are delighted to see you are contributing a new language. That said, the lack of documentation is bad and is due to the fact that right now our highest priority is to get an ink! MVP off the ground. There should totally be some kind of implementer guide in the future for people that want to get started with their own language.
Before diving into your questions let me state something more general: From conceptual stand point we differentiate between kernel/supervisor (pallet_contracts)  and userspace (ABI). The job of the ABI is (among others) to describe calling conventions (encoding of arguments and results). While a contract is not required to conform to the ABI in order to be compatible to pallet_contracts it is strongly recommended to do so when possible because it enables different languages to interact with one another. The reason why this isn't implemented in the supervisor is too allow for flexibility and innovation in the upper layers where iteration is much cheaper. The ABI is the json file describing the contract.
Now to your questions:

Each WASM module should export exactly two functions: call and deploy. Both of these take no arguments and return i32 values (Question: What is the significance of these values?).

None. Those are only there for backwards compatibility reasons and we should really remove support for those. They are completely ignored. The signature you should be using is () -> ().

When a contract is sent a message, the call function is invoked. (Question: How does call determine which message it's been sent? I presume this has something to do with the selector in the contract metadata, but what are the precise mechanics of this?)

A message is a userspace concept. For the pallet_contracts there is only input bytes and output bytes. For our canonical ABI there is a 4 byte selector at the beginning of the input which defines which message is called. But you could totally invent a language which does not use messages.

Is the start section executed for each deploy and each call (the alternative being that the linear memory of the WASM module is snapshotted and restored)? I assume that the answer is yes, but I thought I'd double-check.

Each call or deploy (even nested ones) is executed in its own wasm instance and for each instance the start section is run. There is no memory persistence between instances. You need to use storage manually.

What should go inside of the .contract file?

The .contract file is just the wasm module combined with the metadata.json (which we call ABI) for easier consumption. The referenced hash is blake2.
The code which generates the metadata in the ink! case can be found here here. I am afraid there is no official spec, yet. @ascjones correct me if I am wrong. This is about all I know about the contents of the json file.
How does the selector for each message type manifest in the WASM module? I am guessing that seal_input contains the selector as its first entry, but is that the case?

Why does this need to be declared at all (see previous question)? Is this not an implementation detail?

This is for off chain tooling to be able to inspect the contract. Imagine something like a storage explorer which works on all contracts. Transparency is key when it comes to contracts.

How does the selector for each message type manifest in the WASM module? I am guessing that seal_input contains the selector as its first entry, but is that the case?

Correct. The selector is passed to the contract as input and constitutes the first 4 bytes returned by seal_input.

The seal_XX functions use a pattern of writing their results to a provided output buffer but often do not have a way to prompt them for the amount of data which will be written. What size scratch buffer should be used to ensure that these operations will not overrun?

All functions which write to contract memory have a pointer argument called out_len_ptr which is used by the contract to supply the size of the buffer and is filled by the supervisor with the actual size used on success. This is documented. How big of a buffer you want to provide is up to you. I think ink! uses a 16K buffer for receiving data from the supervisor. Maybe that is something that should be described in the ABI, too (@cmichi @ascjones).
Functions which read values from contract memory forgo a len argument when the size of the encoded value is known at compile time.

hey, we are designing Ask! which is based on assemblyscript, just like ink! for rust.
https://github.com/patractlabs/ask
maybe you can get some useful information for you.

@athei Thank you very much for the detailed response! Apologies for the delay; I have been out of town (and occupied with a bug-squashing spree), but I will get back to this soon and report back on any remaining gaps in getting this proof-of-concept implemented.
@atenjin Thank you for the reference; I will give that a look as well!


[Feature Request] - Add Version to Health Check

This feature would have the health check refer the version of the Gotenberg instance, with an aim to enable integrations to identify requirements and limitations.
{     "status": "up",     "version":"8.2.1",     "details": {         "chromium": {             "status": "up",             "timestamp": "2021-07-01T08:05:14.603364Z"         },         "uno": {             "status": "up",             "timestamp": "2021-07-01T08:05:14.603364Z"         }     } }
Referenced here: https://github.com/paperless-ngx/paperless-ngx/discussions/5820

Previous request / reference from the paperless-ngx maintainer here:
https://github.com/gotenberg/gotenberg/issues/710#issuecomment-1818046685

Hello @latetedemelon,
I don't think that's the goal of the health check endpoint.

The goal of an endpoint is to exchange information.  While you could create
a new endpoint to report system characteristics it seems like extra effort
for no reward.  That’s often why healthcheck type endpoints provide system
status information.
On Thu, Mar 7, 2024 at 01:07 Julien Neuhart @.***>
wrote:

Hello @latetedemelon https://github.com/latetedemelon,
I don't think that's the goal of an health check endpoint to display the
version.
—
Reply to this email directly, view it on GitHub
https://github.com/gotenberg/gotenberg/issues/830#issuecomment-1983037720,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACDYM2JH5EKT5UMI3F45FA3YXAU4BAVCNFSM6AAAAABEJVKBAWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBTGAZTONZSGA
.
You are receiving this because you were mentioned.Message ID:
@.***>


Cask: set write permissions on symlinks

[X] Have you followed the guidelines in our Contributing document?
[X] Have you checked to ensure there aren't other open Pull Requests for the same change?
[X] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

Related to Homebrew/homebrew-cask-versions#6368, turns out some zip files ship with symbolic links set to read-only for the current user, which is not covered by chmod -R (it needs the -h flag which is mutually exclusive).
This pull request adds an extra pass that only touches symlinks on the staged path, courtesy of xargs.
Ping @Homebrew/brew for review, as it blocks the AdoptOpenJDK Cask.

@amyspark I may be missing something here but would macOS not ignore permissions of symlinks anyway?

Under normal circumstances yes @claui. However, the propagation step does not follow symlinks (they can be broken e.g. disk-inventory-x's Frameworks folder). It adds the extended attribute on the symlink itself, which is why it needs the +w permission.

Ping @Homebrew/brew for merging.

Thanks, @amyspark!


False positives for "inheriting from something that is not a class"

The Diagnostics for Inheriting from something that is not a class reports false positives when a library like PonyORM creates classes dynamically.
MWE:
from pony.orm import *
db = Database()
# You're supposed to have your entities inherit from db.Entity
# For reference, this gets created in pony/orm/core.py:746
class MyEntity(db.Entity): # Diagnostic warning because the python language server thinks we're not inheriting from a class
    mystr = Optional(str)

db.bind(provider='sqlite', filename='database.sqlite', create_db=True, create_tables=True)

Need to check what db.Entity evaluates to. Might be unknown which we should be permitting. @MartyBeGood - if it annoys you, you can turn the specific warning off either in settings or by adding #noqa to the line.


Banner Ads Not Vissible For Multiple Pages

The Plugin works fine for a single page. But if you have Multiple Page when the second page loads it covers the entire screen and the ad is not visible.
i use the following to navigate to the second page
frameModule.topmost().navigate("secondPage")

You must call admob again and again, for each page!

I have the same problem with the lastest version.
Any idea?
Thanks


Ghost Recon: Breakpoint runs at 15-20fps

Ghost Recon: Breakpoint runs at 15-20FPS with the Vulkan backend and can even go down to 5FPS if I maximize the window sometimes. The frametime is 50-100ms and it slows down my entire KDE desktop when in Windowed mode. I'm running Arch Linux with Linux 5.16. I've tried mesa-21.3.3 and mesa-git as of a few days ago and there's been no change. This game ran fine in February 2021. I've used all kinds of Wine versions from 6.14 to 7.0-rc5. I've tried DXVK 1.7.11 through 1.9.2 with no change. AMDVLK doesn't seem to work at all with this game, only radeon vulkan.
I am not sure how to grab an apitrace for Vulkan as the D3D11 version of the game doesn't launch with a generic DirectX error. If there is a way, I'd be make a trace.
Software information
Name of the game, settings used etc.
Ghost Recon: Breakpoint, Vulkan renderer. I've tried maximum graphics settings and the minimum with no effect. Fullscreen, borderless, and window mode produces the same issue.
System information

OS: Arch Linux
GPU: Radeon 5700xt
Driver: mesa-21.3.3 and mesa-git
Wine version: tkg-wine 7.0-rc6, wine 7.0-rc5 plain, lutris-6.21, and lutris 6.14
DXVK version: 1.9.2, 1.9.3, and master.

Log files
dxgi, d3d9 and d3d11 log files: https://drive.google.com/file/d/1qG72dpZcoCRDy1hDeeLNzqrkYcbjL8WB/view?usp=sharing

DXVK is not involved when using the native Vulkan version of the game. You'll want to report this to mesa instead.
As for why the D3D11 version of the game does not launch, I don't know, but your d3d11.log is empty so it apparently errors out before even creating a device for some reason.

Thanks! I wasn't sure if DXVK gets invoked when using Vulkan directly.


Adding docs clarifying why TLS config is tied to Gateways

This is a follow up from https://github.com/kubernetes-sigs/service-apis/issues/103.
/cc @jpeach @youngnick
/assign @bowei

LGTM. Let's get this in to get the TLS stuff moving forward.

I'm OK with moving forward with this -- given the discussion that we've had in the past few sessions.
/lgtm
/approve


Fix js interface error identified by fcoleman

Ensure we produce a nil under all circumstances when reading js values.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 82.641%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
14877

Relevant Lines:
18002

💛  - Coveralls


how do you authenticate to HDFS using this plugin?

The plugin currently only supports Pseudo authentication. Option 1 in the following URL https://hadoop.apache.org/docs/r1.0.4/webhdfs.html#Authentication

do you have plans to use TLS/PKI login or username/password for those using cloudera/hortonworks?


Lasb 3501 auto pop init asses notes

What
Link to story
Updated the schema for apply to maat.

It's not ideal us needing to copy across the schemas files from the legal aid schemas repository every time there is a change. Have we explored whether there is a better way of doing this? Ideally the Apply team should publish an artefact for us to consume, but it would also be fairly trivial to write a Gradle task to pull the repo and extract the schemas as part of the build process. It seems like we're duplicating effort here, is this something we can take a look at?

It's not ideal us needing to copy across the schemas files from the legal aid schemas repository every time there is a change. Have we explored whether there is a better way of doing this? Ideally the Apply team should publish an artefact for us to consume, but it would also be fairly trivial to write a Gradle task to pull the repo and extract the schemas as part of the build process. It seems like we're duplicating effort here, is this something we can take a look at?

Fair point @Mhowell494 I'm sure I saw a similar mechanism in one of our other (Ruby?) microservices, maybe it read and updated the schema from here https://github.com/ministryofjustice/laa-criminal-legal-aid-schemas/tree/main/lib/laa_crime_schemas i'm not sure...
I'll take a proper look tomorrow


Release 3.4.0rc2

👍

Coverage remained the same at 82.459% when pulling 72ea4f7457843b3b4418fad2e0fb8cc0125cbd74 on czpython:release/3.4.0rc2 into a468385b5bc1ad13d3a1c16d75257c0d486efb71 on divio:develop.


Latest cordova-res v0.15.0 doesn't work with latest Capacitor v2.2.0

Running cordova-res android --skip-config --copy in an Ionic/Capacitor project fails with
Error: ENOENT: no such file or directory, stat 'resources/android/icon/mdpi-foreground.png'

Steps to reproduce
I have the latest @ionic/cli v6.10.1 and cordova-res v0.15.0 installed globally.
I create a new Ionic React project and enabled Android support with:
ionic start my-app blank --type=react
cd my-app
npm run build
npx cap add android

This installs @capacitor/* v2.2.0.
I then create a resources folder with icon.png and splash.png images, run cordova-res and it fails with the error above. I suspect it may be related to this change.

any updates on this ?

Fixed in 0.15.1. Thanks for the issue!


Non-profit workspace organisation account for Open Source Group KA-Huis

Since we're at this moment an unofficial open source group of the KA-Huis, we sometimes will create documents or other material. In order to make sure we keep this information available in the future, we need to find a service that provides workspace features to store these files. Some options are:

Google Workspace
Microsoft 365

This has been discussed with our stakeholder Enno. The result of discussing this is that if we have files that we would like to keep for future contributors, we can send them to their secretary.


March data dump

Is there any way to get a dump of 6.86 matches only? All I could find were the 500k Dec 2015 and 3.5M dumps.

Large dataset (torrent seeded by @waprin ) and posted to academic torrents

Contains every match played to March 2016
Skill data for most of the matches after May 2015

Small dataset (created from large files by @waprin , these are 4GB samples of the large data set files)
https://storage.googleapis.com/dota-match-dumps/matches_small.csv
https://storage.googleapis.com/dota-match-dumps/player_matches_small.csv
https://storage.googleapis.com/dota-match-dumps/match_skill.csv

We'll probably do another dump in March.
On Tue, Feb 16, 2016 at 1:11 PM, klop<EMAIL_ADDRESS>wrote:

Is there any way to get a dump of 6.86 matches only? All I could find were
the 500k Dec 2015 and 3.5M dumps.
—
Reply to this email directly or view it on GitHub
https://github.com/yasp-dota/yasp/issues/881.

Maybe with skill data this time!
On Feb 16, 2016 10:14 AM, "Albert Cui"<EMAIL_ADDRESS>wrote:

We'll probably do another dump in March.
On Tue, Feb 16, 2016 at 1:11 PM, klop<EMAIL_ADDRESS>wrote:

Is there any way to get a dump of 6.86 matches only? All I could find
were
the 500k Dec 2015 and 3.5M dumps.
—
Reply to this email directly or view it on GitHub
https://github.com/yasp-dota/yasp/issues/881.

—
Reply to this email directly or view it on GitHub
https://github.com/yasp-dota/yasp/issues/881#issuecomment-184806963.

With skill data would be awesome.

do we want to make this a quarterly or semiannual thing?

Pushing back because we're doing import right now.

Posting to say this would be good quarterly (unless you get the BigQuery thing updating live).  Will you post a blog post when the next dump happens?

If it were up to me I'd probably do semiannual but if @albertcui wants to do it quarterly I won't say no (he's the one having to export/upload the data anyway).
Regarding future dumps:
I think at some point after we complete the import we will do a massive pg_dump (this would produce a PostgreSQL-specific dump) with every match ever played (~1.2 billion matches, mostly unparsed).  This will also aid us in doing a data migration if we need to move our match data somewhere else (possibly because of Google getting too expensive).  Then we can do periodic "addendum" dumps to keep updated records exported.  It is up to @albertcui if he wants to continue doing the more generic JSON dumps as well.
ETA for import is 10-15 days.

That'd be great: I'd love to be able to query a db about matches (like the official api, but not limited to the last x hundred games).  If I have to download a massive file first that's not really a problem.
I take it opening up an api of your own would have too high a bandwidth overhead?

Yeah, APIs are expensive to operate.

I'm very interested in using the MMR data for machine learning, is this added in this data dump? I suspect one can estimate a players MMR up to a very high accuracy.

@albertcui are you planning to dump player_ratings?  Or perhaps export a "snapshot" of current MMR data?

I think would be nice if dumps are somewhat synchronized with Majors. This way they could be release in "know" intervals and somewhat related with big updates.

import is done.  Been talking with @albertcui about doing a full dump this time (with every match ever played).
We'd dump matches, player_matches, and player_skill as CSV.

Sounds good

@albertcui I put sample queries in the OP.  You may want to try them locally on your devbox first to make sure they work properly.

yasp=# COPY matches TO PROGRAM 'gzip > /var/lib/postgresql/data/pgdata/matches.gz' CSV HEADER;
ls
COPY<PHONE_NUMBER>
yasp=# COPY match_skill to PROGRAM 'gzip > /var/lib/postgresql/data/pgdata/match_sill.gz' CSV HEADER;
COPY 132447335

matches.gz is 146 GB. Currently exporting player_matches.

Update:
So we encountered some kind of exception while trying to dump player_matches.
Apparently a fix is to perform a vacuum on the table, so @albertcui did this.  It's been running for weeks :(

Started player_matches export again.

COPY<PHONE_NUMBER>6

From @albertcui :
if someone has the hard drive space, ~1TB, if you could help download our data dump and test it, that would be great. https://www.amazon.com/clouddrive/share/2tSLvE98SNMmuv6wwUaPjYLaYA4Rw16ISzW38yAu8yU?ref_=cd_ph_share_link_copy
the large files are split up into 10 GB pieces because of an amazon limitation. You can cat them back together:
http://askubuntu.com/questions/54579/how-to-split-larger-files-into-smaller-parts
@fhoffa , perhaps you could help us do this (since you'll probably want the data to upload to BigQuery?)

If I may make a suggestion for future dumps. Consider doing them incrementally, and ideally split by date. That would make the data much more accessible while also minimizing costs for generation, storage and distribution.

Moving to August, since that's likely when we'll be able to distribute the files (unless someone wants to create a torrent now)

I am slowly but surely downloading the files from the Amazon cloud. I finished matches_split, and am currently downloading player_matches_split. When I finish and verify the files (btw, md5 would help), I will put up a torrent. It moves rather slowly as Amazon closes the session (and the link) after some time, so I have to use a download manager.

I don't know if it's practical or just creates more problems, but i figured i'd throw this out there:
Would it be more practical to do smaller dumps of data but do it regularly? Personally i can't even download such large files, let alone do much meaningful work with it, i'm usually working with between 25-100k matches i've streamed from the JSON, and find that's usually enough for meaningful data.
If that is more practical, i think it could add a lot of value to have these smaller dumps, but regularly, so there's up-to-date data to work with.

Yeah, I think any future dumps will be "incremental" dumps on top of this one.  We don't want to do another one of these full dumps (all together, it ended up costing nearly $800)

Sounds great! Is the plan to do a "500.000 most recent games" dump alongside the full dump, like you did in December? Or will it only be the 1TB file?

Only the big one.  We may just delete the older ones to prevent confusion.
On Jul 1, 2016 12:34 PM, "Kári Gunnarsson"<EMAIL_ADDRESS>wrote:

Sounds great! Is the plan to do a "500.000 most recent games" dump
alongside the full dump, like you did in December? Or will it only be the
1TB file?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/yasp-dota/yasp/issues/881#issuecomment-230030583, or mute
the thread
https://github.com/notifications/unsubscribe/AC_UOIOUEuBSoj0IcfgG8axMh_6Yxn8vks5qRWvXgaJpZM4HbZFE
.

As the 1TB file huge. I think would be nice to keep smaller samples of the database, at least the december 500k matches. It's easier to start to work with the matches.

Started creating .torent file. 508GB gzipped files total.

@nicholashh , can you download and verify the torrent?

@howardchung I am interested in getting the latest data into BigQuery (I work with Felipe, he will help me with BigQuery but he doesn't play dota so more interesting to me). Here are my questinos:

High level summary of the 3 tables and their relationship to what Felipe put in BigQuery last time - https://bigquery.cloud.google.com/table/fh-bigquery:public_dump.dota2_yasp_v1?pli=1

If I want to peek at some rows, is it impossible without downloading the whole thing? Seems like it is because it's gzipped then split up and I think that means you need to cat the whole thing before unzipping but correct me if I'm wrong

Do you still need help hosting or seeding the torrent? Current link looks broken. I will look into my options for doing so.

Thanks, awesome project, hoping to dive into it!

You might want to look at the schema to see what the tables contain (sql/create_tables.sql).  This dump contains a superset of the data in the previous dump.  Last time, we combined the data from multiple tables into JSON match objects.  This time, we opted to dump CSV since it's more flexible.

You could potentially download the first few, streaming unzip it, and process the data you can unzip before it fails.

Albert will be back in about a month.  The data is still hosted in Amazon cloud drive.  We need a torrent made, verified, and seeded.

This is a wonderful set of data, I've got it downloaded and have been poking at it -- in the first segment it seems like a large amount of matches do not have skill levels when I do the join, is this expected?

Yes, we only started collecting skill data in May 2015 and this data contains every match (starting in 2011).  Dotabuff/Dotamax likely have more skill data since they have been running longer but I am not sure if they will do data dumps.

Re: smaller dump, perhaps someone who downloads the full data set can grab the tail end of each of the 3 files and make a "mini" data set.

@xgfs Could you rehost the torrent file? I can put it on academic torrents then we can actually promote the data dump.
Sorry for the delay; I just recently got home and to a stable internet conneciton.
@waprin We'd really appreciate it if you could possibly host/seed the torrent. I probably can't do it until I get my housing situation sorted out in the next few weeks. On a side note, I'm going to be starting at Google on the 8th, so see you soon!

There were some server issues, they are resolved. The link is up again:
http://xgfs.ru/yasp_06_2016.torrent

Thanks, can you set the announce URL to http://academictorrents.com/announce.php ?

Sorry, nevermind I've done it! Thanks a lot!

I've uploaded the torrent to Academic Torrents. Unfortunately, I  won't have a computer with enough disk space to seed for another week :(

File sizes seem too small (player_matches). Can we wc it and confirm the count?
Also it should be March not June. I believe that is the end of the data.

Correct: for some reason ln created a weird half-sized symbolic link (I use the data from the other disk). I will re-create the torrent now. Sorry for the inconvenience. I'll edit the wc to this post as it is ready.

Any chance for that smaller dataset containing recent matches? I don't think I can store the whole 1TB file on my computer and I'd like to be able to work with a more recent dataset than the December one.

Have most of the data downloaded, uploaded it to bucket, still need to import it to BigQuery, will looking into seeding torrent and making smaller dataset.
@albertcui what location are you starting at?

@waprin I'm going to be in MTV.

@albertcui im in MTV, we should get lunch after you're settled
On Jul 29, 2016 3:17 PM, "Albert Cui"<EMAIL_ADDRESS>wrote:

@waprin https://github.com/waprin I'm going to be in MTV.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/yasp-dota/yasp/issues/881#issuecomment-236307820, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAsKVEE6vFotAcwJvaHs_JBHXKlslBc0ks5qanwSgaJpZM4HbZFE
.

Torrent and lengths updated.

Could we get the line counts of the decompressed data?  They should match the COPY output from above.
Also, could you remove _split from the file names?
something like cat match_skill.gz | gunzip | wc -l

@waprin Hi! Any updates on the torrent?

Sorry for big delay, sidetracked by other things.
Byte count on the gzip is the same for matches.gz but not for player_matches.gz. ALso included decompressed sizes:
waprin@unix-instance2:~$ wc -c matches.gz
155941971528 matches.gz

waprin@unix-instance2:/mnt/disks/extra-disk/Downloads$ wc -c player_matches.gz
542664372161 player_matches.gz

waprin@unix-instance2:/mnt/disks/extra-disk/Downloads$ wc -c player_matches.csv
2200436749690 player_matches.csv

waprin@unix-instance2:/mnt/disks/extra-disk/Downloads$ wc -c ~/matches.csv
1253950283105 /home/waprin/matches.csv

I have 4GB truncated files for matches and player_matches as GCS buckets if that helps. Currently putting both small and full dumps into bigquery.
https://storage.googleapis.com/dota-match-dumps/matches_small.csv
https://storage.googleapis.com/dota-match-dumps/player_matches_small.csv
https://storage.googleapis.com/dota-match-dumps/match_skill.csv
Will try to get line count next as well.
I am also now trying to seed the torrent using transmission, not sure if I'm using transmission wrong or nobody else is seeding currently.

If it's being seeded, can we put it on academic torrents and write a release blog post?  @albertcui ?

Moving to September

If it's been seeded can I already download it or its not fully ready yet?

I believe @waprin has the only copy right now.

So all I need is for him to start seeding?

Yes, and I think we need to put it on a tracker (academictorrents)

bumping--any update?  @waprin are you still seeding?  @albertcui can we get it on academictorrents?

(i can also seed it if i have it or if it goes on academictorrents)

Bump again for visibility :)
Moving to October

I was never seeding the torrent because I never had the file downloaded. I could seed the files I downloaded which seems slightly different (byte size) which I got from the Amazon Drive.

Could you check line counts?  If they match the values from the original export I think we can be reasonably sure you have an accurate copy of the data.
Torrents cannot be updated after creation.  You'll have to create a new one.

wc -l is taking forever, not sure if there is a way to parallelize it (it's already decompressed). Either way the matches.gz byte count matches but the player_matches.gz doesn't<PHONE_NUMBER>61 vs<PHONE_NUMBER>41 xgfs reported).

A difference of 1800 bytes seems pretty small.  I imagine it can't be due to incorrectly assembled files since the decompression would have failed otherwise.  It's probably something at the tail end of the file?

I don't know any way to be sure. Are you suggesting I just create a new torrent with the files I have?

Yeah, if you were able to upload them to bigquery etc I think we can be reasonably sure you have valid data.  It would be nice to have the line counts to check, though.

Will try to see if I can figure it out later this week.

@waprin you could use the torrent from academictorrents to verify the file integrity. Just setup the file structure exactly the same, add the torrent pointing to that location and run verify local data/force recheck depending on your bt client.

That file is incorrect and should not have been posted.  @albertcui needs to delete it so we stop confusing people :)

I've deleted the torrent.

@waprin if you can create a new torrent I can upload it to academictorrents. It needs the the announce URL set to http://academictorrents.com/announce.php

And unfortunately I still don't have a computer that I can seed from. My desktop got messed up during the move :(

Updated the OP with the files we are planning to release (large and small datasets)

Fetched the dump from the amazon drive (https://www.amazon.com/clouddrive/share/2tSLvE98SNMmuv6wwUaPjYLaYA4Rw16ISzW38yAu8yU?ref_=cd_ph_share_link_copy) and got these line counts:
==> matches_split.gz.wc <==
1191768404

==> match_skill.gz.wc <==
132447336

==> player_matches_split.gz.wc <==
11720437357(edited)

Waiting for the bit counts. Will post here then they finish. Let me know if you had the same numbers in order to verity the download.

Line counts from postgres (copied from earlier posts):
yasp=# COPY matches TO PROGRAM 'gzip > /var/lib/postgresql/data/pgdata/matches.gz' CSV HEADER;
COPY<PHONE_NUMBER>
yasp=# COPY match_skill to PROGRAM 'gzip > /var/lib/postgresql/data/pgdata/match_sill.gz' CSV HEADER;
COPY 132447335
player_matches
COPY<PHONE_NUMBER>6

It looks like you have one additional line on each one, which is probably just an added newline at the end.  I think you have a valid copy of the data.  If you're able to get a torrent up before @waprin is, we could get it distributed earlier.

@howardchung apparently academic torrents requires an edu address? Any alternatives you can suggest?

You'll have to ask @albertcui , he set up the torrents.  However I don't think you need to create an account, you just need to set the announce URL on your torrent.  I could be wrong though

Yeah I see now he said I just need to set the announce URL when creating the torrent and then he'll upload it. Creating it now and I guess when he comes back we can sort out the details. I am a bit confused what the difference between the announce URL and uploading is. Anyway seems it will take a while to create the torrent anyway so I'll check back in when it's done.

The announce URL I believe sets academic torrents as a tracker. Uploading it to academic torrents is just so that they host the torrent for us. (And I'm not sure if you can use them as a tracker if you don't upload to them? Not super clear on bittorrent technology). Thanks!

@albertcui Fairly certain you have to upload the torrent to academictorrents to use them as a tracker

Perhaps the dataset can be divided into multiple torrents by date?

@klop probably won't happen since someone will need to sort of rows first :frowning_face:

@rossengeorgiev what about by map version?

Any updates here?

It was never clear to me how to upload the torrent. I can set the announce URL but then not sure what to do with it. If someone gives me instructions on how to get it onto a tracker I will do that.

@albertcui can correct me, but you should be able to share the torrent file here like @xgfs did previously, then Albert can put it on academic torrents.  Then you just need to seed it.

@waprin would it be possible to split the various files into smaller pieces? Possibly by line, although that would only be useful for seeding part of the torrent as the lines are not ordered. Either way, having smaller pieces means people can help seed parts without having to download the whole thing. Should make for a healthier torrent.

Perhaps the lines could be sorted by match ID or start_time?  Does BigQuery support exporting data in sorted order?

I don't think Bigquery can do it but I could look into writing a map reduce job to do it or something. I am still just most confused by the actual process of creating the torrent and seeding it, but I'll try to figure it out.
Basic gameplan is to use the top answer here:
http://askubuntu.com/questions/32024/how-to-create-a-torrent-using-the-command-line
Set the announce URL to http://academictorrents.com/announce.php, then upload the torrent file I create here. Let me know if that makes sense.

That sounds correct.  Any update?

Moving to November.

Joined the discord channel and have begun asking questions regarding the best way to get some of the data (All TI6 matches) out of the API into tabular format (currently throwing around the idea of using Postgres to replicate the database and pull it in) . If anyone has a solution to share I would greatly appreciate it.
If this is the wrong place to ask this I apologize. Just trying to get started on a project and OpenDota seems like my best chance at getting it up and running. I would love to begin contributing if there are things that need to be done. Thanks

@waprin any update on seeding?

I have been swamped by work/life but with the holidays coming up I should have some more free time to loop back on this.

Moving to December.  I believe @waprin is sorting the data now so it can be chunked (batches of 100 million match_ids?)

@waprin any update?

@waprin did the match split  complete successfully?
(As an update the player_matches data was split into chunks of 10,000,000 player_matches, resulting in ~119 files).  The plan is to put all the chunks in the same torrent.  Users who don't want the full data set can selectively download certain chunks.

I am still really banging my head against these torrent clients. I am sure I am close to having something working. But I figure the obvious test is to seed a a file from one instance and download in another. I am testing with a small text file and the clients seem to see the files they need to download, but not actually start downloading them.
If anyone has crystal clear permissions on how to seed a file and download it from another using the CLI on Debian/Ubuntu instances using an open tracker, it will simplify things greatly. Otherwise I will figure it out eventually, but that's currently the holdup.

And yeah I split the files successfully and created a torrent, just not very confident others will actually be able to download until I figure out the correct ctorrent or transmission settings.

Maybe you can post the torrent file/magnet link and we can see if we can download successfully?

The torrent itself is 256 MB which seems really big, even given the size of the data. Here is link:
https://storage.googleapis.com/dota-dumps/dota.torrent
I created it by doing
ctorrent -t -u " http://academictorrents.com/announce.php" -s dota.torrent dota_dump
Now, when I do
ctorrent -x dota.torrent
error, "dota.torrent" is really a metainfo file???
error, initial meta info failed.

which is weird, because it can't even read the metadata.
I can also try again by compressing first, but if I compress, then people can't cherry-pick the files as requested (I think).
I tried similarly with transmission-create but have run into different sort of problems.
These are all from GCE instances, I am realizing it's way, way more expensive than just buying a cheap external hard drive, I am traveling for the holidays but I think once I get back I will rebuild a desktop and then maybe be able to seed from there using GUI tools. Until then, I don't mind swinging the cost of seeding from an instance, I just need to figure out how to actually do it.

Also, while we'll eventually get this seeded as a torrent, in the meantime, if a few people are dying for the entire dataset, I can send private links to the GCS buckets with the whole thing to help split the cost between that and the Amazon links. It does seem like the majority of people only want recent or high tier matches anyway.
Speaking of which, are you going to create a dump with recent matches any times?

@albertcui can you help?  256MB seems way too big.  Can you paste your transmission-create command?
No plans to create a new dump at this time.  The API is available for users who need recent data, and based on this effort it takes a lot of time and cost to get a dump out.

Here was my transmission command
transmission-create -o dota-dump.torrent -t " http://academictorrents.com/announce.php" dota_dump/
that creates a 32MB torrent file instead.
https://storage.googleapis.com/dota-dumps/dota_dump.torrent
No idea why there is such a difference in size between ctorrent and transmission.
It also has the same problem where ctorrent fails to read the metainfo.
Understand this is a painful process. Maybe I can run the project myself and start trying to put together dumps, eventually I will understand it and torrents better and the initial pain will be lower. Ideally I'd like to stream new data to the bigquery tables.
Like I said, I'm traveling for the holidays and back early January, then I want to build a new desktop with a big external hard drive. So I'll take another shot at it on that with some GUI tools. So ETA on that would probably be another month.
In the meantime, if Albert or anyone else can offer any insight into what I might be doing wrong trying to create/seed the torrent, I'm happy to try. Like I said, I think just instructions on how to create/seed a tiny file purely from the CLI with a public tracker would be a helpful starting point. I've tried a bunch of tutorials and seem to be getting close, but haven't gotten things quite working.

So doing some quick searching on the subject it looks like the torrent size is based on the number of pieces the data is split into.  If we want a smaller torrent, we probably need to pass an option specifying the number of pieces.  The two commands probably have different defaults.
I assume you have the seeding working as described here? https://lists.freebsd.org/pipermail/freebsd-questions/2009-June/201753.html
Perhaps try another tracker--wasnt there a public one that you were going to try?

Just got too frustrated and wanted to take a break from this, especially since I was traveling. Getting back home tomorrow, going to build a new PC with a bigger disk, will loop back around on this and learn more about torrents sometime this month. I might even try to host my own torrent tracker, might be a good learning experience.

If you just put the blobs on Google Cloud Storage and shared the download links, would you be able to pay for the download bandwidth/storage on your personal account?  Or should we wait until we get a torrent working before making a blog post/public announcement?

I know you guys haven't finish dealing with the original dump, but any chance for a fresh one? Like the last month or something. It would be really useful given the dramatic changes of 700.

unfortunately the old code we used for dumps doesn't work anymore since the move to Cassandra.  No telling when we'll be able to get a new migration script working.
I think @waprin wants to eventually get something set up where match data is directly streamed to BigQuery.  If we get that working then it would probably be the best place to obtain fresh data dumps.

I'm unfamiliar with Cassandra, but there seems to be CAPTURE command that would export the result of queries. Couldn't find any details about performance. Maybe that could do it?
I really like the idea of streaming data to BigQuery, but does seem to be happening any time soon. The original issues was about just a slice of data. I'm looking for the same thing a year later and it seems to be even further from happening.

Depending on how many matches you need, you can use the API to fetch the
match data for randomly selected matches in the time window you want.
On Thu, Jan 26, 2017 at 7:07 AM, Rossen Georgiev<EMAIL_ADDRESS>wrote:

I'm unfamiliar with Cassandra, but there seems to be CAPTURE command that
would export the result of queries. Couldn't find any details about
performance. Maybe that could do it?
I really like the idea of streaming data to BigQuery, but does seem to be
happening any time soon. The original issues was about just a slice of
data. I'm looking for the same thing a year later and it seems to be even
further from happening.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/odota/core/issues/881#issuecomment-275411337, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AC_UOKx2R0dRSLrT0QDA3fIdnFeusA4Oks5rWLawgaJpZM4HbZFE
.

I just ordered a new HDD so I can just seed from home as soon as it arrives. Torrent does seem like the best option.
Would like to stream matches directly from API into Bigquery so will look into that.

I've recently scrapped all dota matches for January and I'm making it available as a torrent. It's 33mil matches without the dark moon ones.
http://static.rgp.io/dota2_matches_jan2017.torrent

Downloaded and created torrent files for player_matches and match_skill from https://github.com/odota/core/issues/881#issuecomment-226920743.
Files:

[x] match_skill.gz 1.60 GB (1643 pieces at 1 MB/piece)
[x] player_matches.gz 505.4 GB (258,763 pieces at 2 MB/piece)
[ ] matches.gz (not uploaded yet)

Downloaded and created torrent files from https://github.com/odota/core/issues/881#issuecomment-226920743. I'll have player_matches up by tomorrow and matches up by as soon as it finishes downloading from Amazon.
Files:

[x] match_skill.gz 1.60 GB (1,643 pieces at 1 MB/piece)
[ ] player_matches.gz (not uploaded yet
[ ] matches.gz (not uploaded yet)

Thanks Discord Nitro.

@albertcui, I believe my links should be good to go as long as you upload the files to academictorrents. It should resolve this error in my client:

Is there anyone seeding the files from @bippum , and is there some sample data available? I've got plenty of space and a home server that can seed 24/7 at 200mbit/s, but I first need to download the data :) . I will leave the torrents provided above running for now, hoping that someone can share them. If there are other torrents available, please let me know.

The OP has small sample datasets.
@albertcui can you please upload the torrents to academic torrents?

I've uploaded matches + match_skill. It won't let me update player_matches:
"Sorry, the piece length is too small. The torrent file must be less than 2MB. Increase your piece length to lower the file size"

I'll attempt to create the torrent again within the day.

Thanks, sorry for the delay. Did uploading the other ones fix the error?

Yes it did.

Great, thanks guys, I'm currently downloading the files and will continue to seed them :). Good work!

Glad to hear you are able to download them OK. I switched torrent clients (from Transmission to Deluge) after having difficulty getting Transmission to do anything, let alone upload. Now I wake up to see that match_skill and matches are seeding! I updated the player_matches link with a 4 MiB piece torrent, @albertcui if you could upload that one to academictorrents? Thanks.

Awesome, once they're all up we can write a release blog post and then we can finally close this! :)

I've uploaded it here: http://academictorrents.com/details/1a0c5736bb54610ad00a45306df2b33628301409

Blog post published:
http://blog.opendota.com/2017/03/24/datadump2/

Hi there guys!
Isn't there anywhere else where the files are available?
I'm currently downloading the "OpenDota - All Matches from March 2016 - Matches"
(matches.gz , 155.94GB)
However the torrent availability is less than 1 (0.781) which means even if I leave this downloading forever, I won't be able to finish the download, cause there are missing pieces in the stream.
(and actually there is only one seeder =/ )

Hi, I'm currently seeding all 3 files with 100% completion, so it should complete eventually.

Hello, would anyone be willing to seed player_matches.gz? Haven't been able to find a seeder for a week now and could really use this data for a project! Thanks in advance!

The copy I had got corrupted during a transfer between hard drives. I no longer have the original files from the amazon cloud drive location, and they aren't obtainable either. Sorry for the inconvenience.

Alright no problem, thanks for the response!
Would anybody else have even a subset of this data available? Perhaps something larger than the 4GB samples above would do the trick.

I wonder if anyone can still seed matches.gz and player_matches.gz? I'm interested in the data for a research project. Thanks.

I am willing to seed it if I can get a copy of it


Add github test workflow

pnpm install, build, test

hey, @birkskyum .
I only see Vitest files in this workflow and I think they pull files from /src

Good catch, it's not needed. Ran i locally just now without the build to make sure, and it's all green.

Thank you


Add welcome command

A command that we can use to point new users to that explains briefly what Smokey is, without pointing to a list of commands.

Coverage decreased (-0.02%) to 65.636% when pulling f7a0fd580cc05a0a09f1decc5813bf5e3a6fdc24 on CalvT:patch-5 into 769811e1929a02a47d68100e82c6b0442311655b on Charcoal-SE:master.


LoadingCache for AuditLogDao objects

Caches AuditLogDao objects for a specific table and expires them after more than 15min without being accessed (executes a create table if not exists statement in constructor).
Adds a depdendency to google guava 18.0.

Thank you.


Tag cloud resources to tmt in Testing Farm

We want to get granular overview of the cloud resources usage on RH ranch.
To distinguish Packit users, we need to override the default tag set to the Testing Farm user name to the specific project.

ok, it does not work, need to talk to Packit

verified it gets to the Testing Farm API and Artemis API \o/

@psss commit message corrected

@psss commit message corrected

Thanks!


Why the handler has direct access to the os.* methods ?

Taking this as an example:

    def ftp_CWD(self, path):
        """Change the current working directory.
        On success return the new directory path, else None.
        """
        # Temporarily join the specified directory to see if we have
        # permissions to do so, then get back to original process's
        # current working directory.
        # Note that if for some reason os.getcwd() gets removed after
        # the process is started we'll get into troubles (os.getcwd()
        # will fail with ENOENT) but we can't do anything about that
        # except logging an error.
        init_cwd = getcwdu()
        try:
            self.run_as_current_user(self.fs.chdir, path)
        except (OSError, FilesystemError) as err:
            why = _strerror(err)
            self.respond('550 %s.' % why)
        else:
            cwd = self.fs.cwd
            assert isinstance(cwd, unicode), cwd
            self.respond('250 "%s" is the current directory.' % cwd)
            if getcwdu() != init_cwd:
                os.chdir(init_cwd)
            return path

Shouldn't os.* calls be hidden behind the self.fs object ? I am developing a virtual ftp server and these calls makes impossible for me to override only the Filesystem object, I need to override almost all the handler to get rid of them.
I think that all those calls should be moved to Filesystem object, don't you think ?

What does your vittual fs's chdir method do exactly? Can you paste the code?
The idea here was using os.getcwd() / os.chdir() directly because no matter what your fs class does the process cwd should never change.

The problem is that the FTP I am implementing is completely virtual and relies on a mysql database to build the user filesystem, here is the chdir:
    def chdir(self, path):
        """Change the current directory."""
        # note: process cwd will be reset by the caller
        assert isinstance(path, unicode), path
        try:
            self.run_api_func(self.api.chdir, path)
        except (OSError, FilesystemError) as err:
            raise err
        else:
            self._cwd = self.fs2ftp(path)

the api calls a rest API that basically checks if the directory exists (FileNotFoundError if not) and if the logged user have permissions.
Furthermore, if my user has an /a/b/c folder, I cannot use os.chdir as it checks for a real path which will not exists on the FTP server :+1:
(venv) ➜  pyftpd (master) ✗ python
Python 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.chdir('/a/b/c')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: '/a/b/c'
>>>

So I think that, for more consistence, the system os* calls should all be moved into the filesystem object, so anyone implementation is really more straightforward:

need to change auth => Authorizer
need to mak ea virtual fs => Filesystem
need more tweaks/callbacks => Handler

What do you think ?

If your fs is virtual then the process cwd will not change therefore
ftp_RETR will also not change cwd back to os.getcwd() because there is no
need.
On Wed, 29 Aug 2018 at 07:07, Giulio Provasi<EMAIL_ADDRESS>wrote:

The problem is that the FTP I am implementing is completely virtual and
relies on a mysql database to build the user filesystem, here is the chdir:
def chdir(self, path):
    """Change the current directory."""
    # note: process cwd will be reset by the caller
    assert isinstance(path, unicode), path
    try:
        self.run_api_func(self.api.chdir, path)
    except (OSError, FilesystemError) as err:
        raise err
    else:
        self._cwd = self.fs2ftp(path)

the api calls a rest API that basically checks if the directory exists (
FileNotFoundError if not) and if the logged user have permissions.
Furthermore, if my user has an /a/b/c folder, I cannot use os.chdir as it
checks for a real path which will not exists on the FTP server 👍
(venv) ➜  pyftpd (master) ✗ python
Python 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.

import os
os.chdir('/a/b/c')
Traceback (most recent call last):
File "", line 1, in
FileNotFoundError: [Errno 2] No such file or directory: '/a/b/c'

So I think that, for more consistence, the system os* calls should all be
moved into the filesystem object, so anyone implementation is really more
straightforward:

need to change auth => Authorizer
need to mak ea virtual fs => Filesystem
need more tweaks/callbacks => Handler

What do you think ?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/giampaolo/pyftpdlib/issues/474#issuecomment-416825717,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAplLMjoGNgnTRZsCxIyxE3BEI8RMowBks5uViGEgaJpZM4WPLKa
.

--
Giampaolo - http://grodola.blogspot.com

Unless I have misunderstood the implementation, I need to keep track of the virtual filesystem (I do keep track by chd into any navigated folder) as when a path is sent to any command (ftp_RETR, ftp_LIST etc...) that path is relative to the current working dir.
Furthermore, my user has a virtual home directory which I don't want him to be able to see, so I use the whole filesystem object to handle this, in the beginning I tried not to use the fs and just call my api directly, but I got some inconsistencies on the ftp client displayed paths (sometimes it showed the fs path, some others the ftp path) so I decided to just use the filesystem object and now it works perfectly.
My point is that, unless there are some drawbacks I didn't think of, any os call should belong in its dedicated filesystem class, it makes custom implementations much easier/cleaner.

Unless I have misunderstood the implementation, I need to keep track of the virtual filesystem (I do keep track by chd into any navigated folder) as when a path is sent to any command (ftp_RETR, ftp_LIST etc...) that path is computed thanks to the CWD.

Correct. The way to "keep track of the cwd" is to set self._cwd attribute in your subclass:
https://github.com/giampaolo/pyftpdlib/blob/13bc5c49b3613d62a0cdf453db6e5ad3e9a6e791/pyftpdlib/filesystems.py#L239
That (_cwd) will be used by ftp2fs method which is called by FTPHandler every time a filesystem related command is involved.
Now that I look at it probably makes sense to add a setter to cwd property which is currently read-only so that users like you can do self.cwd = ... instead of self._cwd = ....

My point is that, unless there are some drawbacks I didn't think of, any os call should belong in its dedicated filesystem class, it makes custom implementations much easier/cleaner.

Not in this case, no (and it's the only one). The fact that os.chdir is used directly from within the handler class is because we don't want to change the process cwd permanently. It's a check which literally reads as "in case the filesystem class, whatever it does, changed process cwd, then reset it to its original". In order to do that we want os.chdir instead of self.fs.chdir because self.fs.chdir may do anything. It may be "virtual" fs, as is your case, and may have no notion of the process cwd, which is what we care about. A direct call to os.chdir is the correct way to deal with that. I'm not sure if I explained myself properly (I hope so =)).
And now that I'm at it, AbstractedFS.chdir should probably use os.listdir(path) or next(os.scandir(path)) instead of os.chdir(path) because os.chdir(path) doesn't play nice with ThreadedFTPServer.

Now that I look at it probably makes sense to add a setter to cwd property which is currently read-only so that users like you can do self.cwd = ... instead of self._cwd = ....

Please dismiss this comment. I now realize cwd is already a setter.

ok I totally see your point, indeed it does make sense ! and in my chdir I do change the internal _cwd pointer to the latest virtual folder, so that's cool.
Thanks for your answers ;)

Prego. ;)