Memory Leak during Massive insert

Hey all,
I've been getting Out of Memory errors when trying to import a largish graph (12 million vertex, 90 million edges) into orientdb using the Java API. With this latest version (I tweaked some settings) of project linked below memory is constantly climbing and  i'm still expecting it to get OOM but processing has not finished yet. I will post results once I get back from work and it finishes running. Any insight you might have would be appreciated thank you!
OrientDB version: orientdb-community-2.0-M3
OS: Windows 7
hardware: 16gb RAM, 200G available disk space
ImportGraph http://pastebin.com/rxhZRCAc
NodeCreator http://pastebin.com/Ne14ec9U
RelationshipCreator http://pastebin.com/nxt9FCnF

In our experience building long chains of vertex-edge-vertex-edge... in java with a variable in memory that holds the beginning vertex will cause (client-side) memory-use to grow unbounded.  We had to modify our code to build the chains in smaller pieces (a few hundred vertex/edges at a time seemed to be the max with 4G available for client), then commit and free the original vertex variable(s) and start a new transaction from the last saved vertex.

Hmm do you think I would be having the same issue even though I am not
using transactions? Also could you clarify on how I would "free up" vertex
variables in my case? They are done one at a time in a method not in a for
loop so I'm unsure how I would do that... Thanks for the quick reply by the
way!
On Jan 13, 2015 8:41 AM, "Keith Freeman"<EMAIL_ADDRESS>wrote:

In our experience building long chains of vertex-edge-vertex-edge... in
java with a variable in memory that holds the beginning vertex will cause
(client-side) memory-use to grow unbounded. We had to modify our code to
build the chains in smaller pieces (a few hundred vertex/edges at a time
seemed to be the max with 4G available for client), then commit and free
the original vertex variable(s) and start a new transaction from the last
saved vertex.
—
Reply to this email directly or view it on GitHub
https://github.com/orientechnologies/orientdb/issues/3381#issuecomment-69763704
.

Hi @GrumpyPants,
did you tried to run with the  orientdb-community-2.0-rc2, since the M3 we fix a few leaks and reduced the overall memory usage, so i think this is already fixed.

Hello tglman,
No I have not tried rc2 yet. I will try that once I get home today and post
results. Thank you.
On Jan 13, 2015 9:10 AM, "tglman"<EMAIL_ADDRESS>wrote:

Hi @GrumpyPants https://github.com/GrumpyPants,
did you tried to run with the orientdb-community-2.0-rc2, since the M3 we
fix a few leaks and reduced the overall memory usage, so i think this is
already fixed.
—
Reply to this email directly or view it on GitHub
https://github.com/orientechnologies/orientdb/issues/3381#issuecomment-69769282
.

Sure, so we had some base vertexes (in a variable in memory) that were the front end of chains connected edge-vertex-edge-vertex... thousands long.  We were creating a committing transactions in chunks, but because we kept the base vertexes in memory the entire chains were also being kept in memory.  So once we "freed" them (i.e. by using a local variable in separate function for each chain rather than a set of variables initialized at the start of a loop) we saw the memory usage on the client go back the familiar saw-tooth pattern.

Just got back from work. The import process is still running from this morning. About 9 hours now. No OOM yet but it is creating edges much slower. Process is consuming 10 gb of ram so far. Looks to be slowly climbing as well? Hard to tell to be honest. I wonder if I were to let it run for the rest of the day if it would finish or if I would get OOM. I'm going to cancel and try with rc-2. Will post results.

8forty,
I believe I understand your scenario now. Am I incorrect in saying that it does not apply to my situation based on how my code is layed out? I'm not keeping any vertices in memory (as far as I know). I'm just directly adding vertices/edges to the graph. I am not using transactions either so while I really do appreciate your help I don't think it will help me in this case.

Process has been running for about 30 minutes now and the memory is steadily rising still. I suspect it would get OOM if I let it run overnight. I am trying to get heap dump for you all but i'm not too familiar on how to do that so I am looking it up now.

@GrumpyPants a thread dump would be useful, and also a dump of memory since this is your problem. You could use the FREE trial of YourKit or any other Java Profiler. With such tools it's quite easy to find who is responsible of keeping object in heap.

Here is a "leak report" from MAT (http://i.imgur.com/1jxQJ7R.png). Is this useful at all? Maybe I didn't generate it correctly. Sorry, I am a little out of my depth if you can't tell haha. I will try and get a thread/ memory dump for you.

Thread dump http://pastebin.com/aHeXx2uZ

Objects with most retained heap http://i.imgur.com/TDz6874.png. Memory is still climbing but at a much slower rate. It has imported 10 out of ~90 million edges. Maybe it will finish this time? Going to let it run over night unless anyone has any other ideas.

How is going after 15 hours? I see 25M of cache objects. Is 25M the number of object or RAM?

No answer so far, i consider this fixed, most of the object showed in the screenshots are associated to caches.
closing

Hey everyone. Sorry for the delayed response. I checked it the following night and it looks like it imported correctly. Memory was indeed slowly climbing the entire time but I guess my memory was big enough that it didn't go out of memory this time with the changes I made I guess. Thanks for all your help everyone.


Suggestion and codefix to add missing cases to switch

TypeScript Version: 3.2.0-dev.20181106
Code
enum E { A, B, C }
function f(x: E) {
    switch (x) {
        case E.A: return "a";
        case E.C: return "c";
    }
}

Expected behavior:
Suggestion on x recommending to add a case case E.B:, and a codefix which does that (perhaps by adding case E.B: throw new Error("todo");).
Actual behavior:
No suggestion.

This also needs to add a break to the last clause if necessary

And it may need to add an import for the enum if necessary


Updates to form examples, inline link and form control components

Summary
Updates to form recipes, form control and inline link components.
🔗 See form example changes in Storybook
🔗 See form control changes in Storybook
🔗 See inline link changes in Storybook
List of notable changes:

updated InlineLink underline to now work across multiple lines.
updated InlineLink dark mode colors to match recent changes to Link
updated FormControl labels from 14px to 16px to match labels in Figma
updated Side by side form example recipes to match latest designs

What should reviewers focus on?

Review the side by side story example, and toggle between default and extended form type controls to see different versions.
Check inline link color values are correct and appear correctly across multiple lines

Steps to test:

🔗 See form example changes in Storybook
🔗 See form control changes in Storybook
🔗 See inline link changes in Storybook
Supporting resources (related issues, external links, etc):

Reference designs in Figma

Contributor checklist:

[ ] All new and existing CI checks pass
[ ] Tests prove that the feature works and covers both happy and unhappy paths
[ ] Any drop in coverage, breaking changes or regressions have been documented above
[ ] New visual snapshots have been generated / updated for any UI changes
[ ] All developer debugging and non-functional logging has been removed
[ ] Related issues have been referenced in the PR description

Reviewer checklist:

[ ] Check that pull request and proposed changes adhere to our contribution guidelines and code of conduct
[ ] Check that tests prove the feature works and covers both happy and unhappy paths
[ ] Check that there aren't other open Pull Requests for the same update/change

Screenshots:
Link colors in dark mode:

 Before  After

Link underline across multiple lines:

 Before  After

New form examples:
https://github.com/primer/brand/assets/13340707/8ad4e1fb-2dde-4b86-9c2c-4895647088a0

@rezrah Could you, please, add link to Storybook?

@rezrah Could you, please, add link to Storybook?

@natalie-iv Links are active in the description now. The deployment takes a while to complete, so worth checking back if they're not there straight away.

@rezrah As agreed in this thread let's keep labels and hints 16px font-size.

Thanks for reviewing this, @natalie-iv 👍

@rezrah As agreed in this thread let's keep labels and hints 16px font-size.

Just to clarify, I've rolled back to 14px based on the most recent conversations in Slack. We'll revise all form components together down the line, including applying the 16px bump to labels. Is that okay @natalie-iv?

One small thing I notices is the wrong border thickness of Verify Button, it should be 1px and it should be without an icon. Could you please check it?

This button would be part of the third party widget we use for security verification. We don't have a way to replicate this properly in our Storybook, so I've intentionally left out of scope. We also don't have that button style in Primer Brand. We'll leave this for eng teams to implement in the final forms.

"Unsubscribe" word here should be also a link (see example https://github.com/enterprise/contact?scid=&utm_campaign=2023q3-site-ww-PremiumSupport&utm_content=Premium+Support&utm_medium=referral&utm_source=github)

I've added a link 👍, but used a placeholder url to avoid linking to our production unsubscribe page.

Message field in the form should not be marked with asterisk (*). Also the phone, but this one I'm checking right now.

Fixed, thank you 👍


The URL in the ? is turned into %3F, causes the return data exception

I have this problem: when I request arguments in URL, such as: http://www.xxxx.com/app/MobileService?action=xxxx. when the request is sent, the URL in the ? is turned into %3F by the url(for target: Target) -> URL method, causing the request to fail.
the method:
private final class func url(for target: Target) -> URL {
        if target.path.isEmpty {
            return target.baseURL
        }

        return target.baseURL.appendingPathComponent(target.path)
    }

The parameters in the url are added to the [String: Any] dictionary, and the work is normal

I have the same question ，have you fix it ?

you can set a good success:
// MARK: - Set the request to the head
let myEndpointClosure = { (target: RequestAPI) -> Endpoint in
print("baseURL:\(target.baseURL)\n path:\(target.path)")

let url = target.baseURL.absoluteString + target.path

print("url:\(url)")
//http://XXXX/api/xlogin.ashx?action=xulogin

//This method will escape special characters
//let url = target.baseURL.appendingPathComponent(target.path).absoluteString
//http://XXXX/api/xlogin.ashx%3Faction=xulogin

let endpoint = Endpoint<RequestAPI>(
    url: url,
    sampleResponseClosure: { .networkResponse(200, target.sampleData) },
    method: target.method,
    parameters: target.parameters,
    parameterEncoding: target.parameterEncoding
)

//Set up your header information
return endpoint.adding(newHTTPHeaderFields: [:])

}
let LoginProvider = RxMoyaProvider(endpointClosure: myEndpointClosure)

public extension URL
{
...
self = URL.init(string: target.baseURL.absoluteString + target.path)!
}


Fix remove ari return value from AutoGMM.fit_predict

Reference Issues/PRs
Fixes #369
What does this implement/fix?
This removes the ari value from the data frame that is returned when running autogmm._fit_cluster. The documentation has also been changed to reflect this modification.

@caseypw don't want to touch dataframe or anything in fit, just want to change return value of fit_predict

@bdpedigo since there isn't a fit_predict function in autogmm.py , I originally modified the _fit_cluster function. I should have instead edited the _fit_for_data function correct?

try checking BaseCluster

How is this @bdpedigo ?

you need to figure out why the diff is so large, should only be a few lines changed. I'm not sure what you did

In base.py, fit_predict and predict now both only return the labels

look under Files changed for this PR. Almost every line looks changed. need to fix that, only lines showing up red or green should be the ones that you changed


Does SSA use asyncgetcalltrace or JVMTI getCallTrace to walkthrough Java stack trace

Team,
Thanks for creating this project. The tools developed were really useful particularly ssa with flamegraphs. But we would like to know whether ssa uses asyncgetcalltrace(AGCT) or JVMTI getCallTrace to walkthrough Java stack trace. Since AGCT output would be accurate in walking through stack asynchronously not impacted by safepoint bias.
Sattish.

Hi,
ssa is using input file. SJK own sampler stcap is using a naive thread dump sampling approach.
Though, ssa can use Java Flight Recorder files. But please keep in mind that you need -XX:+DebugNonSafepoint option enabled for target JVM.
If you are looking for AGCT based profiler take a look at
I would also recommend you to take a look at my article  on topic of sampling bias

Hi,
Thanks a lot for elucidating the queries raised.Appreciate your thought on thread dump sampling


Change title of focused mode view to  Pull Request # and then use the description to be the title of the PR

I actually think the way it currently is (top image) is easier for me to identify what the PR is.


native image -> native executable

Oracle marketing will be changing from "native image" to "native executable" to reduce naming confusion between the native-image tool and the apps it creates.
I'll leave this as a draft PR until the next GraalVM release with the updated documentation.

22.2 is out and as you described Native Executable is everywhere in the docs.
https://www.graalvm.org/22.2/reference-manual/native-image/guides/build-with-reflection/
Thanks for this PR.


[Help wanted] BitmapText goes blurry when scale up to 2

Expected Behavior
I was expected like this when BitmapText scale up to power of two. (2x) (first line glyph 1번째 윤전 돌파!)

Current Behavior
It's goes blurry even using nearest scale mode, have any I missed something?

Possible Solution
Help wanted
Steps to Reproduce
Demo

4.8.9 : https://creffect.com/member/cretapark/pixi-blurry/index.html
6.0.0-rc.2 : https://creffect.com/member/cretapark/pixi-blurry/index-latest.html

Code : https://creffect.com/member/cretapark/pixi-blurry/index.js
Resources

fnt : https://creffect.com/member/cretapark/pixi-blurry/Waffleche.fnt
sheet 1 : https://creffect.com/member/cretapark/pixi-blurry/Waffleche_0.png
sheet 2 : https://creffect.com/member/cretapark/pixi-blurry/Waffleche_1.png

Environment

pixi.js version: 4.8.9, 6.0.0-rc.2
Browser & Version: Chrome
OS & Version: Windows 10
Running Example: See Step to Reproduce's Demo url

Be sure to use roundPixels

The problem is how you're setting scaleMode. I would use the font directly and not use the texture resources from the load. Try this instead:
const app = new PIXI.Application({ width: 320, height: 180 });
document.body.appendChild(app.view);

var face = 'Waffleche';
var fnt = './Waffleche.fnt';

app.loader.add(face, fnt);
app.loader.load(function(loader, resources) {
    const pages = PIXI.BitmapFont.available.Waffleche.pageTextures;
    for (const i in pages) {
        const baseTexture = pages[i].baseTexture;
        baseTexture.scaleMode = PIXI.SCALE_MODES.NEAREST;
        baseTexture.update();
    }
    const content = 'Hello World!';
    const style = { fontName: 'Waffleche', align: 'left'};
    const text = new PIXI.BitmapText(content, style);
    const scaledText = new PIXI.BitmapText(content, style);

    scaledText.scale.set(2);
    scaledText.y = text.y + text.height;
    app.stage.addChild(text, scaledText);
});

@bigtimebuddy Thanks, that's what I expected result!
But it seems pageTextures property aren't in v4, is there equalpment thing about it?
I'm looking in source code of v4 (4.8.9), it's doesn't exposed them in code.
https://github.com/pixijs/pixi.js/blob/359a131eb10485947874b9a48bc2d62c1ee0b1a1/src/extras/BitmapText.js#L557

Oh I see it now, I missed some codes... 🤦
textureResource.texture.baseTexture = PIXI.SCALE_MODES.NEAREST
to
textureResource.texture.baseTexture.scaleMode = PIXI.SCALE_MODES.NEAREST
Thanks for advices, everyone!
I'll close this issue.


Feature: Enable <<inherit>> values

Fixes #60
This PR adds support for de- & encoding resolved and flattened items in Cobbler. It assumes that the caller is aware of the principles and checks the corresponding structures for them.
This PR also adds the previously unexposed flatten and resolved method parameters to the item getters. This is required so that the purely positional arguments in XML-RPC are interpreted by the backend correctly.

If I am not mistaken then the only thing missing is integration testing. @tiltingpenguin do you want me to take #59 into the scope of this PR or do we want to keep that separate from this PR?

I would keep it separate if we are not just testing the code in this PR

@tiltingpenguin Then have fun reviewing. Feel free to critique anything you find. I will gladly extend the commit messages if needed.


Support for Atom?

any plans to support atom in the future?
Thank you

Atom has been archived for 2 years 😅


Why is CPU Checkpointing only available with partitioned activations?

I am trying to better understand offloading activations to CPU (PA_TO_CPU) here https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/runtime/activation_checkpointing/checkpointing.py#L730 and wondering why

"CPU Checkpointing/Contiguous Checkpointing is only available with partitioned activations."

It seems to me like one could be implemented without the other? Or am I missing something.
This is not an issue with the code, just a question to firm up my understanding.
Thanks in advance!
Tagging @jeffra based on git-blame.

Please try the latest version of DeepSpeed.


Transient DBContext no longer working correctly after Update to 3.0

I was updating from 2.2 to 3.0 and using the DI with a transient DbContext like this:
services.AddDbContext<EloContext>(options => options.UseSqlServer(ConfigurationHelper.GetEloConString(config)), ServiceLifetime.Transient);

I am using DI within my constructor, e.g.:
using System.Linq;
using System.Threading.Tasks;
using Microsoft.EntityFrameworkCore;
using SWDi.Base.Models;
using SWDi.DataAccess.Intranet.AutoGenPOCOCode;
using SWDi.DataAccess.Intranet.Repository.Admin.UserModule.Mutation;

namespace SWDi.DataAccess.Intranet.Repository.Admin.UserModule.Query
{
    /// <inheritdoc />
    public class GetUserDataQuery : IGetUserDataQuery
    {
        /// <summary>
        /// Initializes a new instance of the <see cref="GetUserDataQuery"/> class.
        /// </summary>
        /// <param name="context">Intranet Context by DI.</param>
        public GetUserDataQuery(IntranetContext context)
        {
            this.Context = context;
        }

        private IntranetContext Context { get; }

        /// <summary>
        /// Get the Intranet user context using the users (AD) username.
        /// </summary>
        /// <param name="username">The user ID.</param>
        /// <returns>The DB user context.</returns>
        public async Task<User> Execute(string username)
        {
            User u = await this.Context.Users
                .Where(x => x.Username == username)
                .Include(x => x.RefDepartment)
                .Include(x => x.RefUserRole)
                .SingleOrDefaultAsync();

            return u;
        }
    }
}

On my frontpage (build with react) I do several requests on login to fetch the first needed, later cached, data - some of them are accessing the same table at the same time - on 2.2 perfectly fine. Since 3.0 I get an error:
"A second operation started on this context before a previous operation completed. This is usually caused by different threads using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913."
Through git I can pinpoint that to the moment of the 3.0 update.

I am also running into a similar issue. Our application uses scoped contexts via services.AddDbContext without specifying a lifetime. However, we also initialize new DbContexts that are needed for atomic updates or happen in parallel. Each async operation happening in parallel has its own context. Something like this.
public async Task SomeMethod()
{
    var tasks = new List<Task>();
    using (var context = CreateNewDbContext())
    {
        tasks.Add(DoSomething());
    }
    await Task.WhenAll(tasks);
}

private async Task DoSomething()
{
   using (var context = InitializeNewContext())
   {
      context.Add(new Book { Title = "title" });
      await context.SaveChangesAsync();
   }
}

We did not have any context threading issues in 2.2 and am having difficulty getting to the root of the problem.

@Wasseratem @jsheetzati So far I have not been able to reproduce this. Can you please post a small, runnable project/solution that demonstrates the issue. This might require running things in a tight loop to force a concurrency issue.

Also having difficulties reproducing in a sample application at this point. This is leading me to believe we have a different issue in our application :(
Thanks @ajcvickers

EF Team Triage: Closing this issue as the requested additional details have not been provided and we have been unable to reproduce it.
BTW this is a canned response and may have info or details that do not directly apply to this particular issue. While we'd like to spend the time to uniquely address every incoming issue, we get a lot traffic on the EF projects and that is not practical. To ensure we maximize the time we have to work on fixing bugs, implementing new features, etc. we use canned responses for common triage decisions.

Still having this problem, I fought with this for 6 hours today without any idea how to solve it or even reproduce it in a test anvoiriment...

I found something interesting.
I have 3 methods: a lock / an unlock and a get.
The lock writes the userId into a column, the unlock delets it and the get gets the actual data. I have caching problems here, which had lead me again the problem if the transistent context - because of this, as far as I understand, there should be not a single cache: the db context should generated freshly and newly every request I do.
So the lock and the get are within the same WebAPI call, but with the DI the context should be new (2 different classes, db context is inserted in the constructor).
The unlock is a single WebAPI call.
I refreshed my page a few times and clicked on the buttons and see:
Every Lock, every unlock and every get will have the SAME context ID - what?


Investigation: User reported error when submitting reportbacks

BUG
Current Behavior

tell us what happened

Desired Behavior

what should have happened

Steps to Replicate

how can you make it happen every time?

Why This Matters
Countries Affected (optional)
OS/Browser (optional)
Relevant Screenshots + Links
FEATURE OVERVIEW
User Story
Additional Information (optional)
Why This Matters
Current Workaround w/o Feature (if applicable)
HACK DOCUMENTATION
Related Issue
Affected Pages
Hack Explanation
Please note if this needs to be undone at some point and why.

Closing - wrong repo!


Version Bump after September Preview

Now including rush update results for working ci build pipelines.
(please ignore the - tests failures, those pipelines were triggered by too broad of an /azp run command)

/azp run

Note to self: Do not use /azp run because it launches live test pipelines as well...

@danieljurek Please fix the merge conflicts and we can get this merged in

/azp run js - identity

I'm getting build failures locally and in the piplines when running rush rebuild against the repo root... investigating.

This is ready to merge when checks pass.

Seeing a new failure: https://dev.azure.com/azure-sdk/public/_build/results?buildId=124570&view=logs&j=7004824f-51fd-52ae-6e1f-9906ae542efb&t=fac9b094-eaaa-55f4-8b83-7caeb968ea0f&l=3016
FAILURE (1)
================================
@azure/keyvault-certificates (17 minutes 4.1 seconds)
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Writing Node.js report to file: report.20191004.055140.9489.0.001.json
Node.js report completed
 1: 0x9d33e0 node::Abort() [/opt/hostedtoolcache/node/12.11.1/x64/bin/node]
 2: 0x9d4596 node::OnFatalError(char const*, char const*) [/opt/hostedtoolcache/node/12.11.1/x64/bin/node]
 3: 0xb32dbe v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/opt/hostedtoolcache/node/12.11.1/x64/bin/node]
 4: 0xb33139 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/opt/hostedtoolcache/node/12.11.1/x64/bin/node]
 5: 0xcde455  [/opt/hostedtoolcache/node/12.11.1/x64/bin/node]
 6: 0xcdeae6 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/opt/hostedtoolcache/node/12.11.1/x64/bin/node]
 7: 0xcea97a v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/opt/hostedtoolcache/node/12.11.1/x64/bin/node]
 8: 0xceb885 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/opt/hostedtoolcache/node/12.11.1/x64/bin/node]
 9: 0xcee298 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/opt/hostedtoolcache/node/12.11.1/x64/bin/node]
10: 0xcb4bc7 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType) [/opt/hostedtoolcache/node/12.11.1/x64/bin/node]
11: 0xfeaafb v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/opt/hostedtoolcache/node/12.11.1/x64/bin/node]
12: 0x136d539  [/opt/hostedtoolcache/node/12.11.1/x64/bin/node]
Aborted (core dumped)

Looks like an artifact error coming from using the "rerun failed builds" feature

/azp run js - client - ci

/azp run js - core - ci

/azp run js - keyvault - ci


Create Complete Commit API

Changes

Created Complete Commit API that changes the commitStatus of a 'paid' commit to 'completed'.
The endpoint is POST /commits/{commitId}/complete.

Manual Testing Efforts
Success Flow

Also, numCompleted of listing in datastore has increased by 1.

Current commitStatus is not 'paid'

Could you rename edit to update? Haha I know I named it that way before the HOF PR but now that that one is merged, I think update sounds better! :D

@hzxcaryn Okay!

Can you update PR description to add some context about why is this edit commit API needed?

After discussing with @hzxcaryn, decided to change the endpoint to specifically Complete Commit API instead of a more general Edit Commit API. The reason is I thought that RESTful API doesn't allow the use of verbs in the url such as /commits/{commitId}/complete. However, upon looking at commonly used APIs such as this we decided that the use of verb on API url can be allowed if it does a specific operation that is not CRUD. Moreover, the specific endpoint to Complete Commit serves our purpose better as it serves our specific purpose, is clearer and prevents clients from making disallowed edit requests.

Can you update PR description to add some context about why is this edit commit API needed?

@ifhuang It was to be used to complete a commit. Changed the API to Complete Commit API instead after discussion with Caryn, the explanation is in the comment above.


[eventhubs-checkpointstore-table] upgrade dependency @azure/data-tables to ^13.2.2

Packages impacted by this PR
@azure/eventhubs-checkpointstore-table

API change check
API changes are not detected in this pull request.


Fix Glorp / Garage compatibility

I don't think the issues are related. I'll relaunch the CI.

Yes, they are consistently failing in pharo 7 for some reason. I'll merge this issue anyways. I don't believe it is the cause.
Thanks!


[v1.4.0] [Regression] Cannot change height of items in the first row when containerPadding is [0, 0]

Describe the bug
Cannot change height of items in the first row when containerPadding is [0, 0]. Setting containerPadding to [1, 1] fixes the issue.
Your Example Website or App
https://codesandbox.io/s/peaceful-dewdney-4y9gx2?file=/src/ShowcaseLayout.js
Steps to Reproduce the Bug or Issue
Just use the resize handle on an item of the first row, you can't change the height. The only thing I changed in the forked sandbox is to add the containerPadding prop.
Expected behavior
I'm able to resize width AND height of items of the first row
react-grid-layout library version
1.4.0
Operating System Version
Linux
Browser
Chrome
Additional context
No response
Screenshots or Videos
No response

Thx for merging! Will you release a 1.4.1 soon?

Yep, just done.


Unable to execute HTTP request: sts.amazonaws.com

I have installed CB as instructed, but when I try to create my AWS credential I get an error:
cloudbreak_1 | 2015-11-03 15:22:47,537 [http-nio-8080-exec-3] badRequest:42 ERROR c.s.c.c.ExceptionControllerAdvice – [owner:cloudbreak] [type:cloudbreakLog] [id:undefined] [name:cb] Failed to import public key [roleArn:’arn:aws:iam::337….445:role/cb-access’], detailed message: Unable to execute HTTP request: sts.amazonaws.com
cloudbreak_1 | com.sequenceiq.cloudbreak.controller.BadRequestException: Failed to import public key [roleArn:’arn:aws:iam::337….445:role/cb-access’], detailed message: Unable to execute HTTP request: sts.amazonaws.com
I had this a couple weeks ago, but I managed to get past it, but my VM I was using for the Docker images got corrupted, and now I can’t seem to get past this issue this time around.

hi @dbbaskette, looks like the container running cloudbreak-api (creak_cloudbreak_1 container) has no internet connection, or uses a bad dans server. To check which happens:
check network connection
docker exec cbreak_cloudbreak_1 ping <IP_ADDRESS>

check dans server
docker exec cbreak_cloudbreak_1 ping google.com

If Both succeed, than you can check these to command for all other containers..
I suspect, that you do have network connection, but dns fails. If you use cbd on OSX, mostly it happens when you change network while boot2docker is running. In that case please restart it:
boot2docker restart

That helped.  I am on centos, and ANY change to networking including stopping the firewall caused issue.   Once I turned off the firewall with chkconfig and restarted....  i was able to save the cred.  Thanks and this explains why I got past it once (Luck)


Permissions

I was denied permission to push to this repo, can somebody look into/fix that? Adding it to the DSL team might be a good idea.

Fixed.

I still can't push - the repo is listed under the DSL team now but team members only have read permission.

I added you as an administrator to the project Solrun - should give you the necessary privilieges! I also set the DSL team to "write" privileges.

Thanks, now I was able to push.


Create types.constant option

The types.constant option should not be observable. It should be just a typed variable that is stored in the store for internal usage. If there is a better name, we can discuss it.

Solved with 1.3.1


Open File after Save as Excel No Longer opens it via Microsoft Excel

Issue Type: Bug
Hi - after updating to the latest Azure Data Studio, I can no longer open a saved excel file in Microsoft Excel after using the Save as Excel feature on a query. It gives an alert of "An external application is trying to open..." and tries to open the file in Azure Data Studio rather than Microsoft Excel. Please help!
Azure Data Studio version: azuredatastudio 1.33.1 (0b9ef714c823d662815f441aec4a94eaa21940f8, 2021-11-04T17:45:15.872Z)
OS version: Windows_NT x64 10.0.19042
Restricted Mode: No

System Info

Item
Value

CPUs
Intel(R) Xeon(R) E-2286M  CPU @ 2.40GHz (16 x 2400)

GPU Status
2d_canvas: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: enabledopengl: enabled_onrasterization: enabledskia_renderer: enabled_onvideo_decode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
63.85GB (55.53GB free)

Process Argv

Screen Reader
no

VM
0%

Extensions (4)

Extension
Author (truncated)
Version

agent
Mic
0.48.0

query-history
Mic
0.1.0

sql-assessment
Mic
0.6.1

azdata-sanddance
msr
3.3.0

I have the same issue. It's driving me nuts
Version: 1.33.1 (user setup)
Commit: 0b9ef714c823d662815f441aec4a94eaa21940f8
Date: 2021-11-04T17:45:15.872Z
VS Code: 1.58.0
Electron: 12.0.9
Chrome: 89.0.4389.128
Node.js: 14.16.0
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.19042

Same issue - also, occasionally when the popup of "An external application is trying to open xxxx in asuredatastudio" is up for a few seconds, I get another error of "Another instance of azuredatastudio has stopped responding"

Having the same issue
Issue Type: Bug
Pervioulsy opening Excel after saving the query results opened Excel, now Data Studio is tryign to open Excel in editor window
Azure Data Studio version: azuredatastudio 1.33.1 (0b9ef714c823d662815f441aec4a94eaa21940f8, 2021-11-04T17:45:15.872Z)
OS version: Windows_NT x64 10.0.22000
Restricted Mode: No

Thanks for the reports everyone, the fix for this has been checked in. So it should be showing up in the insiders builds starting next week if you wanted to test it out there. Otherwise it will be included in our next release (sometime early December).
Sorry for the inconvenience!

Thank you!
On Fri, Nov 19, 2021 at 11:31 AM Charles Gagnon @.***>
wrote:

Thanks for the reports everyone, the fix for this has been checked in. So
it should be showing up in the insiders builds starting next week if you
wanted to test it out there. Otherwise it will be included in our next
release (sometime early December).
Sorry for the inconvenience!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/azuredatastudio/issues/17618#issuecomment-974353824,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ATB4LJUHRYUNMGXF5FOZ4PDUM2QYLANCNFSM5HP6AWEA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.


Fixed link for Windows

@mebrein mind signing the CLA? :)

Just signed the CLA

Looks like this needs to be rebased.

In the mean time the binaries are updated to 2.8.1. Better cancel this and contact owner to update release script.


'selenium is not installed or configured' handling

https://laboradian.com/operate-browser-on-win-by-python-of-wsl-via-selenium/
https://chromedriver.storage.googleapis.com/index.html?path=84.0.4147.30/
It is necessary to make available to give a specific executable_path from sys.argv[2].
d = webdriver.Chrome(
    executable_path="/mnt/c/Users/foo/tmp/chromedriver_win32/chromedriver.exe"
)

Seleniumくんをやめた


edupont04 from OPS portal: Merge from master to live

Docs Build status updates of commit 452fdbe:
:warning: Validation status: warnings

File
Status
Preview URL
Details

dynamics-gp/payroll/PayrollUS.md
:warning:Warning
View
Details

dynamics-gp/payroll/media/MCADD1.JPG
:white_check_mark:Succeeded
View

dynamics-gp/payroll/media/MCADD10.jpg
:white_check_mark:Succeeded
View

dynamics-gp/payroll/media/MCADD11.jpg
:white_check_mark:Succeeded
View

dynamics-gp/payroll/media/MCADD12.jpg
:white_check_mark:Succeeded
View

dynamics-gp/payroll/media/MCADD13.jpg
:white_check_mark:Succeeded
View

dynamics-gp/payroll/media/MCADD14.jpg
:white_check_mark:Succeeded
View

dynamics-gp/payroll/media/MCADD15.jpg
:white_check_mark:Succeeded
View

dynamics-gp/payroll/media/MCADD2.JPG
:white_check_mark:Succeeded
View

dynamics-gp/payroll/media/MCADD3.JPG
:white_check_mark:Succeeded
View

dynamics-gp/payroll/media/MCADD4.JPG
:white_check_mark:Succeeded
View

dynamics-gp/payroll/media/MCADD6.jpg
:white_check_mark:Succeeded
View

dynamics-gp/payroll/media/MCADD7.jpg
:white_check_mark:Succeeded
View

dynamics-gp/payroll/media/MCADD8.jpg
:white_check_mark:Succeeded
View

dynamics-gp/payroll/media/MCADD9.jpg
:white_check_mark:Succeeded
View

dynamics-gp/payroll/PayrollUS.md

Line 4606, Column 1: [Warning-file-not-found] Invalid file link: 'media/MCADD5.jpg'.
Line 4410, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 4419, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 4955, Column 0: [Suggestion-code-block-indented] Indented code blocks aren't allowed. Use a Markdown code block surrounded by triple backticks (```). NOTE: This Suggestion will become a Warning in late September, 2020.
Line 5972, Column 0: [Suggestion-column-header-missing] Column headers are required for tables. Add appropriate header text. NOTE: This Suggestion will become a Warning in late September, 2020.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


Attach user identification headers to response

Add X-Takahe-User and X-Takahe-Identity headers to response, when available, to allow for better Nginx log enrichment.
Also drop these headers in Nginx so they aren't sent into the world. They probably aren't dangerous since they identfy the users to themselves but strip it for now, just in case.

Could you bundle this into the existing IdentityMiddleware? Each middleware we add involves a small performance hit.

Thanks!


SIMD Compilation issues on mac M1

See https://github.com/RagnarGrootKoerkamp/astar-pairwise-aligner/issues/17 originally:
error[E0425]: cannot find function `_mm_loadu_si128` in this scope
   --> /Users/finn/.cargo/registry/src/index.crates.io-6f17d22bba15001f/editdistancek-1.0.1/src/lib.rs:108:25
    |
108 |                 let x = _mm_loadu_si128(xs.as_ptr() as _);
    |                         ^^^^^^^^^^^^^^^ not found in this scope

error[E0425]: cannot find function `_mm_loadu_si128` in this scope
   --> /Users/finn/.cargo/registry/src/index.crates.io-6f17d22bba15001f/editdistancek-1.0.1/src/lib.rs:109:25
    |
109 |                 let y = _mm_loadu_si128(ys.as_ptr() as _);
    |                         ^^^^^^^^^^^^^^^ not found in this scope

error[E0425]: cannot find function `_mm_cmpeq_epi8` in this scope
   --> /Users/finn/.cargo/registry/src/index.crates.io-6f17d22bba15001f/editdistancek-1.0.1/src/lib.rs:110:25
    |
110 |                 let r = _mm_cmpeq_epi8(x, y);
    |                         ^^^^^^^^^^^^^^ not found in this scope

error[E0425]: cannot find function `_mm_movemask_epi8` in this scope
   --> /Users/finn/.cargo/registry/src/index.crates.io-6f17d22bba15001f/editdistancek-1.0.1/src/lib.rs:111:25
    |
111 |                 let r = _mm_movemask_epi8(r);
    |                         ^^^^^^^^^^^^^^^^^ not found in this scope

For more information about this error, try `rustc --explain E0425`.
error: could not compile `editdistancek` (lib) due to 4 previous errors

you should probably put the SIMD behind a feature flag or so, and/or use neon instructiong (see e.g. https://github.com/Daniel-Liu-c0deb0t/block-aligner/blob/main/src/neon.rs)

Currently, I have simply wrapped the function definition to prevent any compilation errors on M1/M2 architectures. I plan to add support for aarch64 at a later stage.

The rust-bio should use the corrected version by default. Yes, it would be nice if you could do some benchmarks that on real-data sequences. Please, share with me your findings.
first char matches: yes, it sounds reasonable.

thanks for fixing!


-spi n seems broken

I may have spoken too soon on #8 - it stopped working in 28256 mode in my BE6502 when using -spi n (y works).
Also, the transfer time almost triples when using -spi n:

-spi n: Time taken to transfer: 15.22s, fails to boot.
-spi y: Time taken to transfer: 5.27s, boots fine.

Firmware version: 2.0rc8.
Uploader version: 2.0rc8.
macOS 11.2.3
Python 3.9.2

Send me the bin or hex file you are using to test with, I’ll take a look
later on. I did some basic testing last night and was getting good results,
but I relay on reports like yours to iron out the last glitches :)
On Sun, Apr 11, 2021 at 1:32 PM Jacob Krall @.***>
wrote:

I may have spoken too soon on #8
https://github.com/Kris-Sekula/EPROM-EMU-NG/issues/8 - it stopped
working in 28256 mode in my BE6502 when using -spi n (y works).
Also, the transfer time almost triples when using -spi n:

-spi n: Time taken to transfer: 15.22s, fails to boot.
-spi y: Time taken to transfer: 5.27s, boots fine.

Firmware version: 2.0rc8.
Uploader version: 2.0rc8.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/Kris-Sekula/EPROM-EMU-NG/issues/9, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFTKLJH7BNORJBDQG7CQJZLTIIBOPANCNFSM42YBIOOQ
.

Also, can you please paste the output of the console for the failed upload
(with spi -n)
On Sun, Apr 11, 2021 at 1:39 PM Kris Sekula @.***> wrote:

Send me the bin or hex file you are using to test with, I’ll take a look
later on. I did some basic testing last night and was getting good results,
but I relay on reports like yours to iron out the last glitches :)
On Sun, Apr 11, 2021 at 1:32 PM Jacob Krall @.***>
wrote:

I may have spoken too soon on #8
https://github.com/Kris-Sekula/EPROM-EMU-NG/issues/8 - it stopped
working in 28256 mode in my BE6502 when using -spi n (y works).
Also, the transfer time almost triples when using -spi n:

-spi n: Time taken to transfer: 15.22s, fails to boot.
-spi y: Time taken to transfer: 5.27s, boots fine.

Firmware version: 2.0rc8.
Uploader version: 2.0rc8.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/Kris-Sekula/EPROM-EMU-NG/issues/9, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFTKLJH7BNORJBDQG7CQJZLTIIBOPANCNFSM42YBIOOQ
.

python3 /Users/jacob/EPROM-EMU-NG/Software/EPROM_EMU_NG_2.0rc8.py -mem 28256 -spi n -auto n a.out /dev/tty.usbserial-1410

Running EPROM EMU NG python script version 2.0rc8

File used: a.out

Done processing bin

Memory map disabled

Using serial port /dev/tty.usbserial-1410, emulating: 28256 EPROM

<- attempting to get sync
-> HW: v2.1, FW: 2.0rc7, SPI: 1, Auto: 0, Last EPROM: 28256, mygeekyhobby.com 2020 :)

<- Setting "Save to SPI EEPROM" option to disable
-> SPI save OFF

<- Setting "Auto load from SPI EEPROM" option to disable
-> Auto Load OFF

<- processing file

0xB800 >

Time taken to transfer: 15.2s

<- Setting EPROM Type

<- Enable Emulation
-> Emulator Running.

Success, Finished.

a.out.zip

It's macOS 11.2.3

For comparison, the output with -spi y
python3 /Users/jacob/EPROM-EMU-NG/Software/EPROM_EMU_NG_2.0rc8.py -mem 28256 -spi y -auto n a.out /dev/tty.usbserial-1410

Running EPROM EMU NG python script version 2.0rc8

File used: a.out

Done processing bin

Memory map disabled

Using serial port /dev/tty.usbserial-1410, emulating: 28256 EPROM

<- attempting to get sync
-> HW: v2.1, FW: 2.0rc7, SPI: 0, Auto: 0, Last EPROM: 28256, mygeekyhobby.com 2020 :)

<- Setting "Save to SPI EEPROM" option to enable
-> SPI save ON

<- Setting "Auto load from SPI EEPROM" option to disable
-> Auto Load OFF

<- processing file

0xBE00 ACK

Time taken to transfer: 5.26s

<- Setting EPROM Type
-> Bus set to 28256 EEPROM Memory

<- Enable Emulation
-> Emulator Running.

Success, Finished.

I did some testing on my raspberry pi and got the following results:
with "spi -n":
Using serial port /dev/ttyUSB0, emulating: 28256 EPROM
<- attempting to get sync
-> HW: v2.1, FW: 2.0rc8, SPI: 0, Auto: 0, Last EPROM: 2764, mygeekyhobby.com
2020 :)
<- Setting "Save to SPI EEPROM" option to disable
-> SPI save OFF
<- processing file
0xB800 >
Time taken to transfer: 2.97s
<- Setting EPROM Type
-> Bus set to 28256 EEPROM Memory
<- Enable Emulation
-> Emulator Running.
with "spi -y":
Using serial port /dev/ttyUSB0, emulating: 28256 EPROM
<- attempting to get sync
-> HW: v2.1, FW: 2.0rc8, SPI: 0, Auto: 0, Last EPROM: 2764, mygeekyhobby.com
2020 :)
<- Setting "Save to SPI EEPROM" option to enable
-> SPI save ON
<- processing file
0xBE00 ACK
Time taken to transfer: 5.0s
<- Setting EPROM Type
-> Bus set to 28256 EEPROM Memory
<- Enable Emulation
-> Emulator Running.
For the "spi -n" I'm using a new method of transfer (bypassing local buffer
in the Arduino) this "saves" about 0.9s transfer time for the 32kb
filesize...
But looks like there is a difference in behavior for macOS, I'm going to
have to "investigate" ... but first need to dig out my macOS machine :)
In the meantime you can either revert to one of the older versions of
firmware/software or do the following changes to the python script:
Around line 102 add the "buffer_size = 512"
[image: image.png]
And around line 500 add the "data_tx = (":SBN\r\n").encode()
[image: image.png]
This will revert the transfer method for "spi -n" back to "buffered" mode.
Kris
On Sun, 11 Apr 2021 at 16:59, Kris Sekula @.***> wrote:

Great, thanks!
Definitely, some kind of issue, where it says "<- Setting EPROM Type",
there should be a response from the emulator "-> Bus set to 28256 EEPROM
Memory" but I think it got out of sync during the transfer as it shouldn't
have taken 15.2s to transfer ...
I just tested your file with my emulator and script and it was fine. Is it
macOS or Linux you are using?
Kris
On Sun, 11 Apr 2021 at 16:49, Jacob Krall @.***>
wrote:

python3 /Users/jacob/EPROM-EMU-NG/Software/EPROM_EMU_NG_2.0rc8.py -mem 28256 -spi n -auto n a.out /dev/tty.usbserial-1410
Running EPROM EMU NG python script version 2.0rc8
File used: a.out
Done processing bin
Memory map disabled
Using serial port /dev/tty.usbserial-1410, emulating: 28256 EPROM
<- attempting to get sync
-> HW: v2.1, FW: 2.0rc7, SPI: 1, Auto: 0, Last EPROM: 28256, mygeekyhobby.com 2020 :)
<- Setting "Save to SPI EEPROM" option to disable
-> SPI save OFF
<- Setting "Auto load from SPI EEPROM" option to disable
-> Auto Load OFF
<- processing file
0xB800 >
Time taken to transfer: 15.2s
<- Setting EPROM Type
<- Enable Emulation
-> Emulator Running.
Success, Finished.

a.out.zip
https://github.com/Kris-Sekula/EPROM-EMU-NG/files/6293008/a.out.zip
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Kris-Sekula/EPROM-EMU-NG/issues/9#issuecomment-817394935,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFTKLJCFIGDYWHJ45IL5GTLTIIYQZANCNFSM42YBIOOQ
.

I added some debug help to my firmware and programmer script. If the firmware receives a command it doesn't recognize, I added
      Serial.print(rxFrame);
      Serial.println(" not recognized.");

to the bottom of loop().
Then, I added some print debugging to the script:
--- a/Software/EPROM_EMU_NG_2.0rc8.py
+++ b/Software/EPROM_EMU_NG_2.0rc8.py
@@ -521,9 +521,9 @@ try:

                # send to emulator
                ser.write(buff64k[StartAddr:StartAddr+FrameSize])
-               response = ser.readline()
+               response = b" === " + response + b" --- " + ser.readline()^M

-               print("0x%0.4X"%StartAddr,response.decode().strip(), end='\r', flush=True)
+               print("0x%0.4X"%StartAddr,response.decode().strip(),  flush=True)^M

        taken = time.time() - start

Successful output:
…
<- processing file

0x0000 === ->in
 --- ACK
0x0200 === ->in
 --- ACK
0x0400 === ->in
 --- ACK
0x0600 === ->in
 --- ACK
0x0800 === ->in
 --- ACK
0x0A00 === ->in
 --- ACK
0x0C00 === ->in
 --- ACK
…

Unsuccessful:
<- processing file

0x0000 === >
 ---
0x0800 ===  --- >
0x1000 === :DIR not recognized.
 --- 4096 not recognized.
0x1800 === 2048 not recognized.
 ---  not recognized.
0x2000 === 6144 not recognized.
 --- 2048 not recognized.
0x2800 === :DIR not recognized.
 --- 8192 not recognized.
0x3000 === 2048 not recognized.
 --- :DIR not recognized.
0x3800 === 10240 not recognized.
 --- 2048 not recognized.
0x8000 === :DIR not recognized.
 --- 12288 not recognized.
0x8800 === 2048 not recognized.
 --- :DIR not recognized.
0x9000 === 14336 not recognized.
 --- 2048 not recognized.
0x9800 === :DIR not recognized.
 --- 32768 not recognized.
0xA000 === 2048 not recognized.
 --- :DIR not recognized.
0xA800 === 34816 not recognized.
 --- 2048 not recognized.
0xB000 === :DIR not recognized.
 --- 36864 not recognized.
0xB800 === 2048 not recognized.
 --- :DIR not recognized.

Time taken to transfer: 2.07s

So, there's something weird going on with that :DIR command, and only that :DIR command…?
Will keep digging.

ok, great. You don't happen to have a windows or raspberry pi machine
kicking around to try? I want to eliminate the OS from the equation... I'll
try on macOS once I get my old laptop out.
On Sun, 11 Apr 2021 at 17:57, Jacob Krall @.***> wrote:

I added some debug help to my firmware and programmer script. If the
firmware receives a command it doesn't recognize, I added
  Serial.print(rxFrame);

  Serial.println(" not recognized.");

to the bottom of loop().
Then, I added some print debugging to the script:
--- a/Software/EPROM_EMU_NG_2.0rc8.py
+++ b/Software/EPROM_EMU_NG_2.0rc8.py
@@ -521,9 +521,9 @@ try:
            # send to emulator

            ser.write(buff64k[StartAddr:StartAddr+FrameSize])

          response = ser.readline()

          response = b" === " + response + b" --- " + ser.readline()^M

          print("0x%0.4X"%StartAddr,response.decode().strip(), end='\r', flush=True)

          print("0x%0.4X"%StartAddr,response.decode().strip(),  flush=True)^M

  taken = time.time() - start

Successful output:
…
<- processing file
0x0000 === ->in
--- ACK
0x0200 === ->in
--- ACK
0x0400 === ->in
--- ACK
0x0600 === ->in
--- ACK
0x0800 === ->in
--- ACK
0x0A00 === ->in
--- ACK
0x0C00 === ->in
--- ACK
…
Unsuccessful:
<- processing file
0x0000 === >

0x0800 ===  --- >
0x1000 === :DIR not recognized.
--- 4096 not recognized.
0x1800 === 2048 not recognized.
---  not recognized.
0x2000 === 6144 not recognized.
--- 2048 not recognized.
0x2800 === :DIR not recognized.
--- 8192 not recognized.
0x3000 === 2048 not recognized.
--- :DIR not recognized.
0x3800 === 10240 not recognized.
--- 2048 not recognized.
0x8000 === :DIR not recognized.
--- 12288 not recognized.
0x8800 === 2048 not recognized.
--- :DIR not recognized.
0x9000 === 14336 not recognized.
--- 2048 not recognized.
0x9800 === :DIR not recognized.
--- 32768 not recognized.
0xA000 === 2048 not recognized.
--- :DIR not recognized.
0xA800 === 34816 not recognized.
--- 2048 not recognized.
0xB000 === :DIR not recognized.
--- 36864 not recognized.
0xB800 === 2048 not recognized.
--- :DIR not recognized.
Time taken to transfer: 2.07s
So, there's something weird going on with that :DIR command, and only
that :DIR command…?
Will keep digging.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Kris-Sekula/EPROM-EMU-NG/issues/9#issuecomment-817412067,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFTKLJCM2VG4JTKM4CO77QTTIJAN7ANCNFSM42YBIOOQ
.

Still on macOS, will try from a Raspberry Pi next :)
I added functionality to the firmware to print .length() and iterate over the bytes of the rxFrame and found a whole lot of leading NULL bytes:

0x0000 === >
 ---
0x0800 ===  --- >
0x1000 === 20B, ([00000000000000003A444952]) :DIR not recognized.
 --- 4B, ([34303936]) 4096 not recognized.
0x1800 === 4B, ([32303438]) 2048 not recognized.
 --- 871B, ([0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000])  not recognized.
0x2000 === 4B, ([36313434]) 6144 not recognized.
 --- 4B, ([32303438]) 2048 not recognized.
0x2800 === 57B, ([000000000000000000000000000000000000000000000000000003A444952]) :DIR not recognized.
 --- 4B, ([38313932]) 8192 not recognized.
0x3000 === 4B, ([32303438]) 2048 not recognized.
 --- 871B, ([0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000])  not recognized.
0x3800 === 5B, ([3130323430]) 10240 not recognized.
 --- 4B, ([32303438]) 2048 not recognized.
0x8000 === 538B, ([0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003A444952]) :DIR not recognized.
 --- 5B, ([3134333336]) 14336 not recognized.
0x8800 === 4B, ([32303438]) 2048 not recognized.
 --- 91B, ([0000000000000000000000000000000000000000000000000000000000000000000000000000000000000003A444952]) :DIR not recognized.
0x9000 === 5B, ([3332373638]) 32768 not recognized.
 --- 4B, ([32303438]) 2048 not recognized.
0x9800 === 867B, ([000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003A444952]) :DIR not recognized.
 --- 5B, ([3334383136]) 34816 not recognized.
0xA000 === 4B, ([32303438]) 2048 not recognized.
 --- 542B, ([00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003A444952]) :DIR not recognized.
0xA800 === 5B, ([3338393132]) 38912 not recognized.
 --- 4B, ([32303438]) 2048 not recognized.
0xB000 === 84B, ([000000000000000000000000000000000000000000000000000000000000000000000000000000003A444952]) :DIR not recognized.
 --- 5B, ([3430393630]) 40960 not recognized.
0xB800 === 4B, ([32303438]) 2048 not recognized.
 --- 871B, ([000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003A44]) :D not recognized.

Time taken to transfer: 2.42s

<- Setting EPROM Type
5B, ([3433303038]) 43008 not recognized.

<- Enable Emulation
4B, ([32303438]) 2048 not recognized.

Success, Finished.

interesting, do you see those trailing zeros on successful transfer with
spi -y ?
On Sun, 11 Apr 2021 at 18:13, Jacob Krall @.***> wrote:

Still on macOS, will try from a Raspberry Pi next :)
I added functionality to the firmware to print .length() and iterate over
the bytes of the rxFrame and found a whole lot of leading NULL bytes:
0x0000 === >
0x0800 ===  --- >
0x1000 === 20B, ([00000000000000003A444952]) :DIR not recognized.
--- 4B, ([34303936]) 4096 not recognized.
0x1800 === 4B, ([32303438]) 2048 not recognized.
--- 871B, ([0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000])  not recognized.
0x2000 === 4B, ([36313434]) 6144 not recognized.
--- 4B, ([32303438]) 2048 not recognized.
0x2800 === 57B, ([000000000000000000000000000000000000000000000000000003A444952]) :DIR not recognized.
--- 4B, ([38313932]) 8192 not recognized.
0x3000 === 4B, ([32303438]) 2048 not recognized.
--- 871B, ([0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000])  not recognized.
0x3800 === 5B, ([3130323430]) 10240 not recognized.
--- 4B, ([32303438]) 2048 not recognized.
0x8000 === 538B, ([0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003A444952]) :DIR not recognized.
--- 5B, ([3134333336]) 14336 not recognized.
0x8800 === 4B, ([32303438]) 2048 not recognized.
--- 91B, ([0000000000000000000000000000000000000000000000000000000000000000000000000000000000000003A444952]) :DIR not recognized.
0x9000 === 5B, ([3332373638]) 32768 not recognized.
--- 4B, ([32303438]) 2048 not recognized.
0x9800 === 867B, ([000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003A444952]) :DIR not recognized.
--- 5B, ([3334383136]) 34816 not recognized.
0xA000 === 4B, ([32303438]) 2048 not recognized.
--- 542B, ([00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003A444952]) :DIR not recognized.
0xA800 === 5B, ([3338393132]) 38912 not recognized.
--- 4B, ([32303438]) 2048 not recognized.
0xB000 === 84B, ([000000000000000000000000000000000000000000000000000000000000000000000000000000003A444952]) :DIR not recognized.
--- 5B, ([3430393630]) 40960 not recognized.
0xB800 === 4B, ([32303438]) 2048 not recognized.
--- 871B, ([000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003A44]) :D not recognized.
Time taken to transfer: 2.42s
<- Setting EPROM Type
5B, ([3433303038]) 43008 not recognized.
<- Enable Emulation
4B, ([32303438]) 2048 not recognized.
Success, Finished.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Kris-Sekula/EPROM-EMU-NG/issues/9#issuecomment-817416294,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFTKLJEYSINQ2XS7VGMN2CLTIJCJ5ANCNFSM42YBIOOQ
.

nope, it recognizes the :SBN command just fine

It works just fine on my Raspberry Pi Zero, by the way.
~ $ lsb_release -a
No LSB modules are available.
Distributor ID:	Raspbian
Description:	Raspbian GNU/Linux 9.13 (stretch)
Release:	9.13
Codename:	stretch
~ $ python3 --version
Python 3.5.3

(I did have to comment out all of the PySimpleGui code, because of that really old system version of python3)

I tried using /dev/cu.usbserial-* instead of /dev/tty.usbserial-*, per this Stack Overflow question, but saw no difference.
Also, I realized the "slow" send speed of the defect is because ser.readline() is timing out on each block after 0.5 seconds of inactivity 😄

I just tested on macOS as well and it is also broken when I try.
looks like some kind of macOS related issue probably with the pyserial, I
checked with the scope and at some point, I see data stops coming from the
serial port into the Arduino.
lowering the size of the chunk of data that gets sent to the serial port
via the script down to 1024 "fixes" the issue for me (adjust the
buffer_size from 2048 to 1024)
[image: image.png]
Kris
On Sun, 11 Apr 2021 at 18:25, Jacob Krall @.***> wrote:

nope, it recognizes the :SBN command just fine
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Kris-Sekula/EPROM-EMU-NG/issues/9#issuecomment-817419486,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFTKLJCVMT5OEHOVQVRTHOTTIJDZPANCNFSM42YBIOOQ
.

Jacob,
I published rc9 files, changed the transfer size from 2048 to 1024...If I ever figure out how to properly fix it on macOS we can revert to 2048 or even larger but for now to have macOS support working let's stick to 1024.
Let me know if you get a chance to test it.
Kris
P.S. yep, the 15s was the timeout on each block (todo: proper flow control, and timeout handling, for example, the script should have been able to tell you it was not getting the confirmations from the emulator etc. )

rc9 fixes this issue on my Mac 👍

I had a similar problem with another project, also based on the Arduino Nano clone board, which uses the CH340 chip for USB serial communications.
My latest hunch: macOS’ built-in com.apple.DriverKit-AppleUSBCHCOM kext has broken flow control.

Thanks for the info Jacob! Not entirely surprised based on the "quality" of
the driver for those CH340 chips on macOS. It is interesting that the
program "upload" via Arduino doesn't suffer the same issue.
On Fri, 23 Apr 2021 at 20:35, Jacob Krall @.***> wrote:

I had a similar problem with another project, also based on the Arduino
Nano clone board, which uses the CH340 chip for USB serial communications.
My latest hunch: macOS’ built-in com.apple.DriverKit-AppleUSBCHCOM kext
has broken flow control.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Kris-Sekula/EPROM-EMU-NG/issues/9#issuecomment-826027488,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFTKLJATCX5DSIR2OR6JJVLTKI4ANANCNFSM42YBIOOQ
.

It is interesting that the
program "upload" via Arduino doesn't suffer the same issue.

Maybe the "Old" bootloader is fast enough that it doesn't bottleneck?

I upgrade all the nanos that I ship using the ISP connector, so they should
not be using "old bootloader"... it could be a combination of ch340 and
pyserial that is causing the issue, but yeah ultimately looks like some
kind of os level bug...
Kris
On Fri, 23 Apr 2021 at 21:12, Jacob Krall @.***> wrote:

It is interesting that the
program "upload" via Arduino doesn't suffer the same issue.
Maybe the "Old" bootloader
https://github.com/arduino/ArduinoCore-avr/blob/master/bootloaders/stk500v2/stk500boot.c#L761
is fast enough that it doesn't bottleneck?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Kris-Sekula/EPROM-EMU-NG/issues/9#issuecomment-826030869,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFTKLJHTTEOVXGARFIZP523TKJAJ3ANCNFSM42YBIOOQ
.

The “new” bootloader, if I’m reading this right, buffers 256 bytes at a time into RAM:
https://github.com/arduino/ArduinoCore-avr/blob/9f8d27f09f3bbd1da1374b5549a82bda55d45d44/bootloaders/atmega/ATmegaBOOT_168.c#L558

looks like it.
On Fri, 23 Apr 2021 at 22:18, Jacob Krall @.***> wrote:

The “new” bootloader, if I’m reading this right, buffers 256 bytes at a
time into RAM:
https://github.com/arduino/ArduinoCore-avr/blob/9f8d27f09f3bbd1da1374b5549a82bda55d45d44/bootloaders/atmega/ATmegaBOOT_168.c#L558
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Kris-Sekula/EPROM-EMU-NG/issues/9#issuecomment-826036967,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFTKLJFTDWGWRN5TG2CK6TTTKJIBDANCNFSM42YBIOOQ
.


Info length and rate returns different values for different backends

🐛 Bug
torchaudio.info returns the info objects directly from the respective backend. Due to same property naming, users might forget to check how the metadata is calculated. This results in metadata being reported differently depending on which backend is reported.
E.g. sox calculates the length summed across channels whereas soundfile does this per channel (correct)
I would propose to add wrapper for the info objects that - independent of the backend - the most important metadata (length and rate) is identical.
Currently, the sox backend reports a missleading length and the rate parameter is of type float instead of int.
To Reproduce
path =  "any/wavfile.wav"
# soundfile
torchaudio.set_audio_backend("soundfile")
info = torchaudio.info(path)
print(si.length)
print(type(si.rate))

# sox
torchaudio.set_audio_backend("sox")
info = torchaudio.info(path)
print(si.length)
print(type(si.rate))

Expected behavior
soundfile reports the correct metadata, sox should be corrected so that:
# sox
torchaudio.set_audio_backend("sox")
info = torchaudio.info(path)
print(si.length // si.channels)
print(int(si.rate))

Environment
torchaudio==0.5.0 from pypi

Relates to #236?

I agree that both backend should return the same values. We need to make sure we document the change of either so that users are aware of any changes. Thoughts?

I could reproduce this;
code
import torchaudio

path = "test/assets/vad-go-stereo-44100.wav"

# soundfile
torchaudio.set_audio_backend("soundfile")
si, _ = torchaudio.info(path)
print(si.length)
print(type(si.rate))

# sox
torchaudio.set_audio_backend("sox")
si, _ = torchaudio.info(path)
print(si.length)
print(type(si.rate))

43339
<class 'int'>
86678
<class 'float'>

I checked #236, and it does not fix the issue.
I think it's more appropriate to handle it on Python side, as suggested.
(Copy data from sox backend C++ struct to SignalInfo Python class and apply the fix there.)
That way, info function returns the same class regardless of backend.

That way, info function returns the same class regardless of backend.

we had already used the sox backend in production (before soundfile was merge in), so we made the modifications to info in our code (should have reported earlier!). If we change the info results now this might cause unexpected results for users. I would rather solve this by

clarify documentation on  torchaudio.info(...), stating that this returns the native info object of each backend and each contain a different number and definitions of metadata (e.g. some metadata also depends on the codec/container being used).
create a new torchaudio info object that holds just the most common important infos: sample_rate,  nb_samples and nb_channels. No idea how to name that, though...

we had already used the sox backend in production (before soundfile was merge in), so we made the modifications to info in our code (should have reported earlier!). If we change the info object now, this might cause unexpected results for users. I would rather solve this by

You mean that you adjusted your code for sox, soundfile, or both?

You mean that you adjusted your code for sox, soundfile, or both?

I had nb_samples = si.length // si.channels calculated for sox. With the introduction of the soundfile backend, I was required to correct this.
I looked into the unit tests of the torchaudio.info and I guess the error was not caught since the unit tests only [covers mono signals]https://github.com/pytorch/audio/blob/master/test/test_io.py#L247) (where the correction is correct). Can we add a stereo (and possibly multichannel) regression test for info?

I looked into the unit tests of the torchaudio.info and I guess the error was not caught before as the unit tests only cover mono signals (where sox and soundfile give the same output). Can we add a stereo (and possibly multichannel) regression test for info?

@faroit Sure. We should add the test. Let us know if you would like to open a PR. If you are busy I will add it to my backlog.

There are two issues here.

"When doing torchaudio.info() on a multi channel audio file, the length is multiplied by however many channels there are." as in #236.
sox and soundfile backend return similar object, but with different values.

For the first, I'm concerned about such change for backward compatibility for sox. I would recommend instead adding a key nb_samples = si.length // si.channels to the object returned in python.
For the second, I second adding a stereo test for consistency of course. :) One of the two would need to be updated to match the other. Since sox has been there longer, soundfile's si.length is the one that should be adjusted. A warning when someone invokes info with soundfile would need to be raised, and the documentation aligned.
I prefer this solution to creating a new abstraction, simply since it keeps us closer to the intended set up so far.
Thoughts? Do you want to open pull requests for this?

Regarding the first issue, this is probably a very old issue in audio vocabulary and maybe it would be good to make a formal distinction between frames and samples as it's done in libsndfile. Over there:

A frame is just a block of samples, one for each channel.

and

...seeks are defined in number of (multichannel) frames. Therefore, a seek in a stereo file from the current position forward with an offset of 1 would skip forward by one sample of both channels.

which makes totally sense to me (also soundfile is the defacto standard when it comes to proper handling of audio i/o. However this would probably lead to too many changes here but it makes sense to put the definition that is used here ("we define samples are the number of frames in an audio signal per channel").

For the second, I second adding a stereo test for consistency of course. :) One of the two would need to be updated to match the other. Since sox has been there longer, soundfile's si.length is the one that should be adjusted. A warning when someone invokes info with soundfile would need to be raised, and the documentation aligned.

I agree this is probably the simplest solution

Do you want to open pull requests for this?

I started with a new test #639 that is expected to fail and can propose a fix for this as well ( in the same PR?)

@faroit
I am working on a new backend with sox and along the way, I addressed most of the issues with the existing sox backend.
https://github.com/pytorch/audio/pull/726
https://github.com/pytorch/audio/pull/728
Unfortunately the new backend is not compatible with the existing one due to TorchScript's technical limitation and design choice, but you may want to check out once it lands.

@mthrok
has this been addressed in #761 and can be closed?

@mthrok
has this been addressed in #761 and can be closed?

Hi @faroit
#761 is for the new "sox_io" backend and the original "sox" backend is not affected. We plan to deprecate and decommission "sox" backend in the future, by replacing the default backend with the new "sox_io" backend. I was thinking that we can close this issue when we decommission the "sox" backend.

@mthrok

#761 is for the new "sox_io" backend and the original "sox" backend is not affected.

does sox_io match the metadata format of soundfile then? If not, I would still propose to unify the defintion of samples, and frames with respect to the duration of a file.

In 0.7.0 release, we introduced the new interface for soundfile backend and in https://github.com/pytorch/audio/pull/978 we switched the default backends which report the number of samples correctly. This will be a part of 0.8.0 release.
The legacy backends are scheduled to be removed in 0.9.0.

Closing the issue as the new backends handle this properly.


tileset.modelMatrix not working for .i3dm with ModelExperimental enabled

Found this while testing for #10250. With ExperimentalFeatures.enableModelExperimental = true, the instanced model doesn't show up at the correct position, instead remaining on the ground. The PR itself did not introduce it; I tested on main and the same behavior occurs.

Here's a sandcastle to demonstrate.

Not sure if it's the same problem, but I noticed in this sandcastle that the i3dm models seem to be oriented incorrectly:
The trees are knocked down:

I need to check if the bridge is an i3dm, but it also seems to be oriented incorrectly:

Some more details:

I did a git bisect, it pointed me to this commit. It seems like when we refactored the model matrices to support model size, we didn't take .i3dm into account
Looking back at the instancing code, I'm finding it rather confusing. i3dm handles world space and the RTC transform differently than EXT_mesh_gpu_instancing so there's a separate shader code path (see LegacyInstancingStageVS.glsl). I wonder how much overlap there is. If needed, we could split InstancingPipelineStage into two if there's not much overlap.
I notice that the old Instanced3DModel3DTileContent computes a model matrix with the RTC offset baked in before passing it to ModelInstancingCollection. The instancing stage doesn't include this offset in the matrix passed to the shader
I also notice that InstancingPipelineStage has a u_instance_nodeTransform matrix which includes handling the axis correction. I wonder if this matrix isn't quite correct with the other matrix changes

Interesting, if I set u_instance_nodeTransform() to return Matrix4.IDENTITY things look correct.

Not quite... it doesn't seem to make a difference for the cubes example. Sounds like this will require some more digging to better understand the matrices in the instancing code.

These are the expected orders of transformations
i3dm: computed tile transform * RTC center * instance transform * model matrix * y-up-to-z-up * glTF node hierarchy * vertex
gltf: computed tile transform * model matrix * y-up-to-z-up * glTF node hierarchy * instance transform * vertex

(Computed tile transform includes the tileset's model matrix and the tile transforms)

I'm starting to understand this in bits and pieces.
I've been taking a closer look at the above equations and trying to translate them to the current code's variables. Let me try to organize my thoughts so far:
Tile = "computed tile transform" = tile.computedTransform
Rtc = "RTC Center" = components.transform (poorly named! this is only ever used for RTC)
Instance = "instance transform" = TRS from instancing parameters = getInstancingTransform() in shader stage
Model = "model matrix" = sceneGraph.modelMatrix
Zup = "y-up-to-z-up" = applied via ModelExperimentalUtility.correctModelMatrix()
Node = "gltfNode hierarchy" = runtimeNode.transformToRoot * runtimeNode.transform

Let me re-write the desired transforms a little bit:
let Local = Model * Zup * Node // local transform for this node

i3dmModel = Tile * Rtc * Instance * Local
gltfModel = Tile * Rtc=Identity * Local * Instance

Introducing Identity to take the place of the Rtc transform, then it's more obvious that the only difference is where to apply the Instance transform.
That said, there's some gotchas here:

in the glTF case, Tile * Rtc * Local can be used directly as drawCommand.modelMatrix which is used in czm_modelView However, in the i3dm case, you have to keep Local separate since Instance comes from an attribute and must be handled in the shader. This is why the old code had a legacy instancing stage.
I'm noticing that for 3D Tiles we pass Tile to model.modelMatrix, which means Tile = Model. It might be necessary to make more of a distinction in the code.

Some other potential problems in the current code:

I think due to the confusing name of components.transform, I thought this was applied like Model * components.transform, but really it should go on the left.
runtimeNode.transform's meaning changed recently, so now u_instance_nodeTransform = runtimeNode.transform * Zup is only using a single node's matrix, not the whole chain to the root node.  Plus it's applying Zup incorrectly according to the previous comment.

I have to think a bit about the cleanest way to fix these problems.

Hm, looking at the old code for 3D Tiles 1.0 formats, it was a bit different:

for b3dm, the content passes in Model = Tile * Rtc.
for i3dm, ModelInstanceCollection.modelMatrix = Tile and then for a specific instance, Model = Rtc * Instance. So I think the equations become:

let Components = components.transform
let Model = Tile * Components
let Local = Zup * Node // local transform for this node

i3dmModel = Model * Instance * Local
gltfModel = Model * Local * Instance

Furthermore, I found at least one case where components.transform is used for more than the RTC transform (see PntsLoader.js) so I'll start with just documenting the property better.

Update: I tried an experiment:
uniformMap.u_instance_nodeTransform = function () {
  return Matrix4.multiply(
    renderResources.runtimeNode.transformToRoot,
    renderResources.runtimeNode.axisCorrectedTransform,
    new Matrix4());
};

and that seems to fix things. What was happening is the glTF root node had a z-up-to-y-up transform that was getting ignored, so the axis correction was not being canceled out properly.
This still doesn't address the tileset.modelMatrix problem with the cubes example. That's probably something else... I'm surprised, since that should be folded into the computed tile transform.

Regarding the tileset.modelMatrix issue, my understanding of the non-I3DM path is that when we change the model matrix of the model (by multiplying the tileset/tile.modelMatrix, the change gets applied to the model matrix that is attached to the DrawCommand. That is then set to the czm_model/czm_modelView in UniformState  and accessed in the shader. Since the legacy instancing path uses, u_instance_modifiedModelView which is based on sceneGraph.components.transform - if that remains unchanged, then any change to the model matrix will not have an effect. So, perhaps the thing to try would be to make sure the world space transformation of the model is being propagated to u_instance_modifiedModelView.


Pulls

#13500

@rhusar / @pferraro Is this the PR supposed to address the kind of failures coming up in the "Linuc MicroProfile - JDK 8" job or is that a different one?
Not sure if that job needs kicking off again or not.

@darranl That's a different one, the WFLY-13869 doesn't seem to have resolved the issue completely.

This was just me testing #13500 again against an updated master. The CI jobs that were running at the time on #13500 were against an outdated master.


Fix install on newer ubuntu distros.

This tweaks the script logic so that the fallback works for newer ubuntu distros rather than being for the older distros. This should increase the likelihood that deps.sh works automatically on new ubuntu distros without any changes.

Superseded by #488.


chore: switched to Node@16 (NPM 8)

refs 80768

nvm install 16.13.0
nvm alias default 16.13.0
nvm use 16.13.0

Or simply: nvm use
Reasons:

We wanted to switch to Node16 anyway as we would like to stay on a newer local node version.
We would like to tryout NPM workspaces. Lerna is not maintained anymore.

https://github.com/lerna/lerna/issues/2975
https://github.com/lerna/lerna/issues/2703

A full re-installation of dependencies is 37 seconds faster with Node 16.
Not a lot, but with #418 we will roughly reduce it by another minute.


etcd-1.0.5 build failures with http-conduit-2.3

Seen on the Stackage build server; I was also able to reproduce the errors locally like so:
stack unpack etcd-1.0.5 && cd etcd-1.0.5
edit stack.yaml # add the following stack.yaml
stack build

# stack.yaml
resolver: nightly-2018-03-10
extra-deps:
- http-conduit-2.3.0

The errors were as follows:
/Users/dan/scratch/etcd-1.0.5/src/Network/Etcd.hs:226:31: error:
    Variable not in scope:
      withManager
        :: (Manager
            -> m0 (http-client-0.5.10:Network.HTTP.Client.Types.Response
                     ByteString))
           -> IO
                (http-client-0.5.10:Network.HTTP.Client.Types.Response ByteString)
    |
226 |     body <- responseBody <$> (withManager $ httpLbs req)
    |                               ^^^^^^^^^^^

/Users/dan/scratch/etcd-1.0.5/src/Network/Etcd.hs:239:31: error:
    Variable not in scope:
      withManager
        :: (Manager
            -> m1 (http-client-0.5.10:Network.HTTP.Client.Types.Response
                     ByteString))
           -> IO
                (http-client-0.5.10:Network.HTTP.Client.Types.Response ByteString)
    |
239 |     body <- responseBody <$> (withManager $ httpLbs $ req { method = "PUT" })
    |                               ^^^^^^^^^^^

/Users/dan/scratch/etcd-1.0.5/src/Network/Etcd.hs:248:31: error:
    Variable not in scope:
      withManager
        :: (Manager
            -> m2 (http-client-0.5.10:Network.HTTP.Client.Types.Response
                     ByteString))
           -> IO
                (http-client-0.5.10:Network.HTTP.Client.Types.Response ByteString)
    |
248 |     body <- responseBody <$> (withManager $ httpLbs $ req { method = "POST" })
    |                               ^^^^^^^^^^^

/Users/dan/scratch/etcd-1.0.5/src/Network/Etcd.hs:257:31: error:
    Variable not in scope:
      withManager
        :: (Manager
            -> m3 (http-client-0.5.10:Network.HTTP.Client.Types.Response
                     ByteString))
           -> IO
                (http-client-0.5.10:Network.HTTP.Client.Types.Response ByteString)
    |
257 |     body <- responseBody <$> (withManager $ httpLbs $ req { method = "DELETE" })
    |                               ^^^^^^^^^^^

I've removed etcd from the Stackage nightly build plan, and unfortunately it won't appear in the initial LTS 11 release.
To get back into nightly, open a PR on the Stackage repo as usual. To get into LTS 11, open an issue on https://github.com/fpco/lts-haskell/issues.


Playwright - Favorite/Unfavorite Parent

Description
The following PR includes:

Refactored test 003
New test 004 to favorite and unfavorite a parent DAO
Adjustments to local worker total count; 6 instead of 8 by default
Adjustments to Github Actions build time 🤞

Testing
npm run tests

Docker tweaks from this were pulled into https://github.com/decent-dao/fractal-interface/pull/576, closing this one for now, since testing the components will need to change significantly once the UI is updated.


[BUG] the github link in header is redirected to old invalid repository

I ll make it to redirect to this repository.
Please assign  this issue to me under GSSOC'24

already done


Cluster up failure because kube-controller-manager is grumpy with GCE

Almost thought it was related to #8196, but it just showed similar symptoms. One of our internal probers failed to bring up a cluster because the node wasn't reporting healthy. It's a 1-node cluster, and kubectl get nodes shows as empty. We see a lot of the following in the
E0513 21:35:57.860505       8 nodecontroller.go:247] Error syncing cloud: Get https://www.googleapis.com/compute/v1/projects/cloud-kubernetes-prober/zones/us-central1-f/instances?alt=json&filter=name+eq+k8s-prober-2015-05-13t14-32-12-node.%2A: oauth2/google: can't get a token from the metadata service; not running on GCE

Meanwhile, node-1 is trying to dial the master and getting rebuffed:
E0513 21:34:27.925099    2961 reflector.go:130] Failed to list *api.Node: Get https://k8s-prober-2015-05-13t14-32-12-master.c.cloud-kubernetes-prober.internal/api/v1beta3/nodes?fieldSelector=metadata.name%3Dk8s-prober-2015-05-13t14-32-12-node-1: dial tcp <IP_ADDRESS>:443: connection refused

cc @davidopp @cjcullen (who I think saw something similar).
cc @dchen1107 (don't think it's a node issue this time, though).
I have the cluster up still if someone wants to poke around.

Is there a way to reproduce this problem?

I have the cluster up still if someone wants to poke around.

It's actively grumpy.

The node controller uses GCE MIG introspection to find the set of node machines to add, so it's not surprising that if it can't get an access token, that the cluster doesn't have any nodes.
The question is: why can't it talk to the metadata server?
And what should we do when this happens?
This looks like a GCE problem, can you ssh into the machine and try to get the token manually?
If that doesn't work, I'd ping someone on the VMM team to diagnose what is hosed in the machine.
I don't think this is P0, as a) it doesn't happen that often, and b) I'm not sure if there is a fix our our side.

this is very stale at this point, probably not worth leaving open


Lock unused Material attributes in Maya

Lock/Hide material attributes in Maya that our engine doesn't use.
Also, consider unlocking these attributes when the plugin unloads (if necessary).

It has been decided that it is not a good idea to lock functionality from other plugins.


ridiculous fill prompt at every new line prompt in the debugger console

It looks like this:

I get it over and over again and have to backspace to get rid of it before I can start typing whatever I'm trying to type.

Thanks for report. We fixed very similar bug in next version(hope it will be on this or next week).


Namespace typo

https://github.com/davidrevoledo/ServiceBus.LogginPlugin/blob/def057b3647e67d9edd9cbf6d72e750876b363cf/src/ServiceBus.LogginPlugin/Infraestructure/MessageExtensions.cs#L30
Most likely because of the containing folder.

Fixed here, great thanks.
https://github.com/davidrevoledo/ServiceBus.LogginPlugin/commit/db713ba40b6647cb1681243d441cb01d5f4043a0


feat(website): New landing page hero and footer

refs #5962

p.s. I invited your @firezone email to Vercel. That will allow you to leave live comments on the preview in case that's useful for collab

@Patticatti Merging


Update Steve affiliation

I updated my affiliation.  I replaced the Anaconda entry, since they are no longer an institutional partner.
There are still a few institutions that should be included but are currently not listed, but I didn't want to assume an icon for them.

@blink1073 I didn't know about the new gig, congrats Steve :-)

Thanks, I start today :tada:


Bump inmanta-dev-dependencies[async,core,pytest] from 2.106.0 to 2.107.0 (#7166)

Pull request opened by the merge tool on behalf of #7166

Merged in 6450fe7b367068343c0b86961e467a51a8844586


Building Swift-Docc and adding a Makefile to make it easier

Description
This pull request comes with additional functionality and pre-built documentation to do a Swift-Docc build of ATProtoKit. Additionally, there is a Makefile and markdown file for easy Swift-Docc builds.
The built site is located in the docs folder. Therefore, if you actually want to publish the site on Github Pages, please change the settinggs to build from the /docs folder as shown in the following picture.

I have not done many pull requests, so I apologize if I am not very good at this.
Linked Issues
#[6]
Type of Change

[ ] Bug Fix
[x] New Feature
[x] Documentation

Checklist:

[x] My code follows the ATProtoKit API Design Guidelines as well as the Swift API Design Guidelines.
[x] I have performed a self-review of my own code and commented it, particularly in hard-to-understand areas.
[x] I have made corresponding changes to the documentation.
[x] My changes generate no new warnings or errors in the compiler or runtime.
[x] My code is able to build and run on my machine.

Info
Checking that the framework works is done with a build of the framework itself and with our own project atpctrl, which depends on the framework.
In both cases, we have already confirmed that the build was successful.
Screenshots (if applicable)

Document
Additional Notes

I added OSLog's Logger to the code, but removed it once from discussion in Issues.#[7]
Added a Makefile for easy build of Swift-Docc and a markdown file on how to build it.

Credits
If you want to be credited in the CONTRIBUTORS file, you can fill out the form below. Please don't remove the square brackets.

Name: [Keisuke Chinone]
GitHub: [KC-2001MS]
Bluesky: [iroiro-work.bsky.social]
Email<EMAIL_ADDRESS>

Thanks for the pull request. I'll review it shortly.
I should note that the way you inputted the "Linked Issues" section. You marked it as "#6" instead of just #6. This is more of an issue with how I wrote it, however, so I'll simply mark it here and I'll update the pull request template.
One the pull request is merged, #6 will close automatically.


Makka / 1794 use quantity and allowance props

Description
Replace amount prop with quantity and allowance props.

Remove amount prop, add quantity & allowance.
Initially allowance is 0.0, so for the allowance we use approvalValue || quantity
After the transaction is approved, we can then use the allowance prop instead as this is populated with the allowance value.


add support for global event hooks, closes #5

Per our conversation, this PR adds support for defining event listeners for global events in a choo-store event object.
Global events do not get added to state.events[storeName] (since they are already reserved under state.events) and do not get added to the actions object (since they are not namespaced events).
Will update the README in a followup commit.

Perfect! This is exactly how I imagined implementing this. Thanks so much @s3ththompson!

This has been released as 1.0.0 since the module's fairly stable now and I've been using it in production for over a month now.
@s3ththompson I've invited you to be a collaborator, thank you for your contribution!

This feature came in very handy today!
Thanks again for making it happen @s3ththompson 🙏


cob: Use hashes for operation ids

See commits
Closes #243
Supersedes #339

I can't seem to find where we test this scenario of [revision, revision, merge] === [revision, merge, revision]. Did you add it and if so could you point it out to me? :)

Ah, I added a different set of tests here: https://github.com/radicle-dev/heartwood/blob/af6ffce7de983f5dec980a2bb4a6233e82d5e62c/radicle/src/cob/thread.rs#L632-L692
I can add a more specific test in patch too though.

Ah, I added a different set of tests here:

Ya, I don't think that captures the mutating phenomenon since all the comments are the same.

All done.
I made the revision handling change to identity, patch and thread COBs, and added some tests. I think we may want to move to an LWWMap eventually, but need to explore this.
I also created a EntryBlob type to hold the oid/data.