Plugin log levels not respected

What steps did you take and what happened:
Wrote a plugin and tried to use log levels other than Info.
What did you expect to happen:
When the ark --log-level option is set to debug, my plugin's Debug/Debugf logs should be printed.
The output of the following commands will help us better understand what's going on:
N/A
Anything else you would like to add:
Never seeing this line printed: https://github.com/heptio/velero/blob/1675943f44dfec16f369d631046e52f6f252283c/pkg/cmd/server/plugin/plugin.go#L41
which infers that: https://github.com/heptio/velero/blob/1675943f44dfec16f369d631046e52f6f252283c/pkg/cmd/server/plugin/plugin.go#L34 is not pulling in log levels from anywhere.
Inside: https://github.com/heptio/velero/blob/7e4fca428d73899cbbdb00da8d340d44984975d8/pkg/plugin/logger.go#L28, we can see that the log level isn't referenced so the default: Info is inferred.
Environment:

Velero version (use ark version): v0.10.1
Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:04:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes installer & version: Minikube v0.30.0
Cloud provider or hardware configuration: vSphere VM
OS (e.g. from /etc/os-release): Ubuntu 18.04

thanks for the report @josephgorse, we have an in-flight PR #1049 to fix this!


Support API mocking via MSW

Is your feature request related to a problem? Please describe.
When writing a web application that makes many API calls, it is often challenging to get good code coverage without mocking the API.
Describe the solution you'd like
The Mock Service Worker library can intercept requests on the network level and mock responses in a framework-agnostic way, and it runs on Node.js too.
MSW can already be used with Connect, but it requires mocking on the HTTP level, without any benefits that the schema provides: For example, it is easy to misspell a header name or request path - even though both are already well-defined by the schema or protocol.
It would be fantastic if there was a type-safe integration for MSW with Connect to remove the boilerplate, for example:
import { setupWorker } from 'msw'
import { service } from '@connectrpc/connect-msw'

const worker = setupWorker(
  service(ElizaService, {
    say(req) {
      return { sentence: `you said ${req.sentence}` }
    }
  }),
)

worker.start()

Describe alternatives you've considered
@connectrpc/connect-playwright provides API mocking with Playwright, but that means all tests must be written in Playwright, limiting the options.
Additional context

Prior effort for gRPC-web: mswjs/msw#238 and a blog post.
Ongoing effort to use fetch API primities in MSW: mswjs/msw#1436

I like the basic API, the one thing I'd like to figure out is how to apply a baseUrl. Ideally we'd only have to define it once (in the transport) but there's no real good way to extract that. So I think the second arg might need to be an options object:
const worker = setupWorker(
  ...service(ElizaService, { baseUrl: "/api" }, {
    say(req) {
      return { sentence: `you said ${req.sentence}` }
    }
  }),
)

Though it's not quite as attractive an api, and it gets worse with multiple services:
const worker = setupWorker(
  ...service(ElizaService, { baseUrl: "/api" }, {
    say(req) {
      return { sentence: `you said ${req.sentence}` }
    }
  }),
  ...service(BigIntService, { baseUrl: "/api" }, {
    add() {
      return { result: 2n };
    }
  )
)

Though maybe that's fine. I thought of pushing the options arg to the last, but the service impl will always be the most important argument so I feel like it shouldn't be stuck in the center of argument list.

I should have been more clear that the code snippet was really just a pseudo-code example to illustrate that users don't have to hand-write requests paths and serialization code - a pretty nice benefit from the schema.
We'll certainly need to accept several options (the baseUrl you mention, but also serialization options, and likely also server side interceptors from #527). I'm sure that a reasonable API will manifest, even if it might need a couple of iterations 🙂

MSW can already be used with Connect, but it requires mocking on the HTTP level
Hi @timostamm, do you have any examples using this approach? Thank you very much!

@nguyenyou, the Connect protocol is very simple, especially for unary and JSON. This should mock the rpc Say of the demo service Eliza:
import { http, HttpResponse } from 'msw'

export const handlers = [
  http.post('connectrpc.eliza.v1.ElizaService/Say', async (request) => {
    const requestJson = await request.json();
    return HttpResponse.json({
      sentence: `you said: ${requestJson.sentence}`
    })
  }),
]

(Based on the current examples on mswjs.io)

@timostamm
Hi. I have a question regarding the proposed solution you provided.
I tried it with my code, and MSW is not intercepting the requests.
Could this maybe be related to me using<EMAIL_ADDRESS>?
This is how I call my RPC method:
const transport = createConnectTransport({
  baseUrl: `http://<IP_ADDRESS>:3003/grpc`,
  httpVersion: '1.1',
});
const client = createPromiseClient(ExampleService, transport);
return client.hello({ name: 'bob' });

This is how I tried mocking it (I tried all possible URL combinations, because I thought the problem was there, but now I'm not sure anymore):
const mswServer = setupServer();
mswServer.listen({ onUnhandledRequest: 'error' });

const handler = http.post(
  'http://<IP_ADDRESS>:3003/grpc/martines3000.example.v1.ExampleService/Hello',
  // eslint-disable-next-line @typescript-eslint/require-await
  async (request) => {
    console.log('request', request);
    return HttpResponse.arrayBuffer(
      new HelloResponse({ message: 'Does not work' }).toBinary(),
      { status: 200 },
    );
  },
);

mswServer.use(handler);

The handler code is a little bit different, but the problem is that it never gets to this part (MSW never intercepts the request).
Do you maybe have any ideas where I should look ?
Thanks for the help!

I have a

@timostamm Hi. I have a question regarding the proposed solution you provided.
I tried it with my code, and MSW is not intercepting the requests. Could this maybe be related to me using<EMAIL_ADDRESS>?
This is how I call my RPC method:
const transport = createConnectTransport({
  baseUrl: `http://<IP_ADDRESS>:3003/grpc`,
  httpVersion: '1.1',
});
const client = createPromiseClient(ExampleService, transport);
return client.hello({ name: 'bob' });

This is how I tried mocking it (I tried all possible URL combinations, because I thought the problem was there, but now I'm not sure anymore):
const mswServer = setupServer();
mswServer.listen({ onUnhandledRequest: 'error' });

const handler = http.post(
  'http://<IP_ADDRESS>:3003/grpc/martines3000.example.v1.ExampleService/Hello',
  // eslint-disable-next-line @typescript-eslint/require-await
  async (request) => {
    console.log('request', request);
    return HttpResponse.arrayBuffer(
      new HelloResponse({ message: 'Does not work' }).toBinary(),
      { status: 200 },
    );
  },
);

mswServer.use(handler);

The handler code is a little bit different, but the problem is that it never gets to this part (MSW never intercepts the request).
Do you maybe have any ideas where I should look ?
The PR #830 implementation will probably solve all of this issues and making mocking much easier, right ?
Thanks for the help!

I got it working with MSW 2.0 using the #830 PR's util method like this:
export const handlersTest = service(
   UnitKeywordService,
   {
   	baseUrl: 'http://<IP_ADDRESS>:23012',
   },
   {
   	queryKeywords: () => {
   		return {
   			response: new CarrelSearchResponse(),
   			designId: 1011,
   		}
   	},
   }
)

@timostamm Hi. I have a question regarding the proposed solution you provided.

@martines3000
I got it working with MSW 2.0.3 using the #830 PR's util method like this in Jest test set up.
import { setupServer } from 'msw/node'

export const handlersTest = service(
   UnitKeywordService,
   {
   	baseUrl: 'http://<IP_ADDRESS>:23012',
   },
   {
   	queryKeywords: () => {
   		return {
   			response: new CarrelSearchResponse(),
   			designId: 1011,
   		}
   	},
   }
)

export const mockServer = setupServer(...handlersTest)

beforeAll(() => mockServer.listen())

You can find the service method here:
https://github.com/connectrpc/connect-es/blob/9c384c0e9ccdfa59722b122f4699a383a320b8a7/packages/connect-msw/src/service.ts#L9

@martines3000, yes, the PR will make this much easier and less error-prone. We'll pick it up again soon.
You're using createConnectTransport from @connectrpc/connect-node, which is using the Node.js built-in modules http and http2. msw can only intercept requests that use fetch() for HTTP requests.
You have two options in this case:

Use createConnectTransport from @connectrpc/connect-web instead for your clients. It will use fetch() for HTTP requests, and that works very well on recent versions of Node.js.
Continue to use createConnectTransport from @connectrpc/connect-node, but use the router transport for testing instead of msw. It's documented here.

I think this is actually an important point. We have to be very clear that MWS only intercepts fetch(). That's not self-explanatory just from reading the MSW docs or this issue description.

@timostamm Hi. I have a question regarding the proposed solution you provided.

@martines3000
I got it working with MSW 2.0.3 using the #830 PR's util method like this in Jest test set up.
import { setupServer } from 'msw/node'

export const handlersTest = service(
   UnitKeywordService,
   {
   	baseUrl: 'http://<IP_ADDRESS>:23012',
   },
   {
   	querySomething: () => {
   		return {
   			something: new SampleSomething(),
   		}
   	},
   }
)

export const mockServer = setupServer(...handlersTest)

beforeAll(() => mockServer.listen())

You can find the service method here:
https://github.com/connectrpc/connect-es/blob/9c384c0e9ccdfa59722b122f4699a383a320b8a7/packages/connect-msw/src/service.ts#L9

However, I do have a related issue using the above method, which completely confuses me.
When I use the exact same set up in Storybook with its msw add-on
https://github.com/mswjs/msw-storybook-addon/issues/121#issuecomment-1777192208
The mocked return type errs with a ConnectError","message":"[deadline_exceeded] the operation timed out in the browser.
And if compare the request received by Jest version of createResponseResolver and the one received by storybook msw-addon setupWorker createResponseResolver, the timeout header are different:

With default timeout ms = 12345m

export const grpcWebTransport = createGrpcWebTransport({
	baseUrl: 'http://<IP_ADDRESS>:23012',
	useBinaryFormat: true,
	defaultTimeoutMs: 12345,
})

const headerGrpcTimeout = request.headers.get('grpc-timeout')
console.log('headerGrpcTimeout', headerGrpcTimeout)

the Jest resolver receives:

`() => date.getTime() - Date.now()m`. Sometimes `() => void m`

while the storybook msw-addon setup receives the correct `12345m`

I don't know why this happens. Cause MSW add-on doesn't seem to do anything different, and both are using the exact same client and unary method call. But somehow one's GrpcTiemout header is incorrect.

Thanks  @timostamm for the quick reply and the solution. It works now :100: .
Also thanks @boan-anbo for the suggestion, will try it out.

I faced the same issue and found the root cause of the problem
https://github.com/mswjs/msw/issues/2049
So in can be easily fixed in the connect-node package by replacing
import * as http from "http";
import * as https from "https";
import * as http2 from "http2";

with
import http from "http";
import https from "https";
import http2 from "http2";

But I'm not sure about the correct way to create a PR with these changes.
Is there some contributing.md?

I finally got a chance to play with v2 of protobuf-es and was looking about to see if there was any information on using msw with it. I stumbled across this thread and the #830 PR and it helped me solve a few issues I was running up against. Would love to see an official @connectrpc/connect-msw at somepoint!
I'm not sure if its the correct way to go about it (as I suspect information about the transport/configuration should probably be taken into account), but figured I'd share what I ended up with as it seemed to work ok in my small example project.
Util
I originally planned to create a factory function that took a baseUrl and transport and returned a another curried factory that could then be used to create handlers for that endpoint (using just the service, method and response). I didn't get that far though and also didn't see a way to avoid having to pass in the schema too.
/** @/test/createStubConnectHandler.ts */

import type { DescService, MessageInitShape, MessageShape } from '@bufbuild/protobuf';
import type { GenMessage } from '@bufbuild/protobuf/codegenv1';
import { ConnectError, createConnectRouter, type ServiceImpl } from '@connectrpc/connect';
import { http, HttpHandler, HttpResponse } from 'msw';

// Types
type ServiceMethod<S> = S extends DescService ? keyof S['method'] : never;
type ResponseSchema<S, M extends ServiceMethod<S>> = S extends DescService ? GenMessage<MessageShape<S['method'][M]['output']>> : never;
type OutputMessage<S, M extends ServiceMethod<S>> = S extends DescService ? MessageInitShape<S['method'][M]['output']> : never;

type StubConnectHandlerFactory = <
  S extends DescService,
  M extends ServiceMethod<S>,
  T extends ResponseSchema<S, M>,
  O extends OutputMessage<S, M>,
>(options: { baseUrl: string, service: S, method: M, schema: T, response: O | ConnectError }) => HttpHandler;

// Helpers
const AsyncIterator = {
  fromStream: async function* <T>(stream?: ReadableStream<T> | null): AsyncGenerator<T> {
    if (stream === undefined || stream === null) return;
    const reader = stream.getReader();
    try {
      while (true) {
        const { done, value } = await reader.read();
        if (done) return;
        yield value;
      }
    } finally {
      reader.releaseLock();
    }
  },
  toStream: function <T>(input?: AsyncIterable<T>): ReadableStream<T> | undefined {
    if (input === undefined || input === null) return;
    const stream = new ReadableStream<T>({
      async start(controller) {
        try {
          for await (const chunk of input) controller.enqueue(chunk);
        } finally {
          controller.close();
        }
      },
    });
    return stream;
  },
};

// Factory
export const createStubConnectHandler: StubConnectHandlerFactory = (options) => {
  // Build URL
  const endpoint = `${options.baseUrl}/${options.service.typeName}/${options.method.toString()}`;

  // Create Mock Handler
  return http.post(endpoint, async ({ request }) => {
    // Copy Request Parts
    const { url, body: bodyStream, method, headers: header, signal } = request.clone();

    // Create Body AsyncIterator
    const body = AsyncIterator.fromStream(bodyStream);

    // Create RPC Handler
    const handler = createConnectRouter()
      .service(options.service, {
        [options.method]: async () => {
          if (options.response instanceof ConnectError) throw options.response;
          return options.response;
        },
      } as Partial<ServiceImpl<any>>)
      .handlers[0];

    // Run RPC Handler
    const result = await handler({ httpVersion: '2.0', url, method, header, body, signal });

    // Respond
    return new HttpResponse(AsyncIterator.toStream(result.body), { status: result.status, headers: result.header });
  });
};

Usage
Here are some example unary response handlers showing both success and failure. Its pretty clean to work with and the type-inference/guarding is really nice.
/** @/test/handlers.ts */

import { Code, ConnectError } from '@connectrpc/connect';
import { SystemService, GetSystemInfoResponseDtoSchema } from '@example/sdk';

import { createStubConnectHandler } from '@/test/createStubConnectHandler';

export const getSystemInfoHandlerSuccess = createStubConnectHandler({
  baseUrl: 'http://localhost:8080',
  service: SystemService,
  method: 'getSystemInfo',
  schema: GetSystemInfoResponseDtoSchema,
  response: {
    version: '1.0.0',
  },
});

export const getSystemInfoHandlerFailure = createStubConnectHandler({
  baseUrl: 'http://localhost:8080',
  service: SystemService,
  method: 'getSystemInfo',
  schema: GetSystemInfoResponseDtoSchema,
  response: new ConnectError('Not Authenticated!', Code.Unauthenticated),
});

Configuring msw and vitest
I was using<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>and configuration and test cases looked a like this...

show code
/** @/test/server.ts */

import { setupServer } from 'msw/node';

import { getSystemInfoHandlerSuccess } from '@/test/handlers';

const handlers = [
  getSystemInfoHandlerSuccess,
];

export const server = setupServer(...handlers);

/** @/test/setup.ts */

import { server } from '@/test/server';

beforeAll(() => {
  server.listen({ onUnhandledRequest: 'error' });
});

afterEach(() => {
  server.resetHandlers();
});

afterAll(() => {
  server.close();
});

/** @/src/example.test.ts */

...
import { server } from '@/test/server';
import { getSystemInfoHandlerFailure } from '@/test/handlers';
...

it('should return system info when authenticated', async () => {
  // Setup
  const request = create(GetSystemInfoRequestDtoSchema, {});
  const expected = create(GetSystemInfoResponseDtoSchema, { version: '1.0.0' });

  // Test
  const actual = await client.getSystemInfo(request);

  // Check
  expect(actual).toStrictEqual(expected);
});

it('should throw an error if not authenticated', async () => {
  // Setup
  const request = create(GetSystemInfoRequestDtoSchema, {});
  const expected = new ConnectError('Not Authenticated!', Code.Unauthenticated);
  server.use(getSystemInfoHandlerFailure);

  // Test
  const actual = client.getSystemInfo(request);

  // Check
  await expect(actual).rejects.toThrow(expected);
});


Does Caffe2 really use GPU when run a model translated from caffe?

Hi,
I met some problems when I did tiny-yolo detection in different platform. Firstly, I  run tiny-yolo  on the original darknet. Then, I used  caffe-yolo to convert it to Caffe. Finally, I use caffe_translator.py to convert Caffe model of tiny-yolo to Caffe2 model. My CPU chip is intel i7 and GPU is GTX970.
I found that,
With darknet, it costs 1.563 seconds per image in CPU mode, and only 0.003 seconds in GPU mode.
With caffe, it costs  0.471 seconds per image in CPU mode, and only 0.041 seconds in GPU mode.
Obviously, it can be seen that the speed of using GPU is much faster than CPU mode in darknet and Caffe.
However, when I test tiny-yolo in Caffe2 on two different hardware platform (one has GPU device, and the other doesn't have. Since Caffe2 detects hardware automatically and I cannot manually choose mode I wanna use ), both of them costs about 0.25 seconds.
When I opened the NVIDIA X Server to see the GPU Utilization, I found that,  in darknet GPU mode, the GPU Utilization can reach 30%, and in caffe and caffe2 GPU mode, the GPU Utilization can only reach 10%.
Since the speed of Caffe GPU mode is 10 times faster than CPU mode, it can be proved that GPU really did work. But in Caffe2, the speed of CPU mode and GPU mode are almost same, and  GPU Utilization is very low.
I doubt that,  did the GPU really work when I ran a Caffe2 model converted from Caffe? Why there is no difference in speed when I ran CPU and GPU mode in Caffe2?
Thanks a lot for your help!

Caffe2 works with GPU - but not in workspace.Predictor framework.
You must use older way to setup/invoke your networks.
Example YOLO net from your post showed me for 3 fullHD images 4.6 seconds in CPU mode - and 0.05 seconds in CUDA mode:
a=np.random.rand(2,3,1080,1920).astype(np.float32)

workspace.ResetWorkspace();
#device_opts = core.DeviceOption(caffe2_pb2.CPU, 0)
device_opts = core.DeviceOption(caffe2_pb2.CUDA, 0)

init_def = caffe2_pb2.NetDef()
with open("exec_net.pb", 'rb') as f:
    init_def.ParseFromString(f.read())
    init_def.device_option.CopyFrom(device_opts)
    workspace.RunNetOnce(init_def.SerializeToString())

net_def = caffe2_pb2.NetDef()
with open("predict_net.pb", 'rb') as f:
    net_def.ParseFromString(f.read())
    net_def.device_option.CopyFrom(device_opts)
    workspace.CreateNet(net_def.SerializeToString())

name = net_def.name
out_name = net_def.external_output[-1];
in_name = net_def.external_input[0]

#%%
workspace.FeedBlob(in_name, a, device_opts)
print ('Running net..."' + name + '"' )

tic = time.time()
workspace.RunNet(name, 1)
results = workspace.FetchBlob(out_name)
tm = time.time() - tic
print (tm)

Hi diz-vara,
I tried your approach and ran into couple of issues.  Any idea what might be the issue ?
Error1 :
Code
with open("predict_net.pb", 'rb') as f:
net_def.ParseFromString(f.read())
net_def.device_option.CopyFrom(device_opts)
workspace.CreateNet(net_def.SerializeToString())
Error
Required modules imported.
BATCH SIZE:  1
Config set!
Functions set.
Mean was set to:  128
INIT_NET =  /local/caffe2/build/caffe2/python/models/bvlc_alexnet/init_net.pb
PREDICT_NET =  /local/caffe2/build/caffe2/python/models/bvlc_alexnet/predict_net.pb
found  /local/caffe2/build/caffe2/python/models/bvlc_alexnet/init_net.pb ...now looking for  /local/caffe2/build/caffe2/python/models/bvlc_alexnet/predict_net.pb
all needed files found! loading the model in the next block
Number of batches:  1000
Pre-processing images...
Done pre-processing, initializing net...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1004 16:05:53.260538 10063 workspace.cc:157] Blob data not in the workspace.
Traceback (most recent call last):
File "alexnet_pretrain.py", line 202, in
workspace.CreateNet(net_def.SerializeToString())
File "/local/caffe2/build/caffe2/python/workspace.py", line 167, in CreateNet
StringifyProto(net), overwrite,
File "/local/caffe2/build/caffe2/python/workspace.py", line 190, in CallWithExceptionIntercept
print("Traceback for operator {} in network {}".format(op_id, net_name))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xb7 in position 164: ordinal not in range(128)
Error2
Code
When i changed the code to
with open("predict_net.pb", 'rb') as f:
net_def.ParseFromString(f.read())
net_def.device_option.CopyFrom(device_opts)
workspace.CreateNet(net_def)
Error
Required modules imported.
BATCH SIZE:  1
Config set!
Functions set.
Mean was set to:  128
INIT_NET =  /local/caffe2/build/caffe2/python/models/bvlc_alexnet/init_net.pb
PREDICT_NET =  /local/caffe2/build/caffe2/python/models/bvlc_alexnet/predict_net.pb
found  /local/caffe2/build/caffe2/python/models/bvlc_alexnet/init_net.pb ...now looking for  /local/caffe2/build/caffe2/python/models/bvlc_alexnet/predict_net.pb
all needed files found! loading the model in the next block
Number of batches:  1000
Pre-processing images...
Done pre-processing, initializing net...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1004 16:13:15.168926 17579 workspace.cc:157] Blob data not in the workspace.
Traceback for operator<PHONE_NUMBER> in network bvlc_alexnet
Traceback (most recent call last):
File "alexnet_pretrain.py", line 203, in
workspace.CreateNet(net_def)
File "/local/caffe2/build/caffe2/python/workspace.py", line 167, in CreateNet
StringifyProto(net), overwrite,
File "/local/caffe2/build/caffe2/python/workspace.py", line 186, in CallWithExceptionIntercept
return func(*args, **kwargs)
RuntimeError: [enforce fail at operator.cc:52] blob != nullptr. op Conv: Encountered a non-existing input blob: data

@diz-vara   I see " YOLO net from your post showed me for 3 fullHD images 4.6 seconds in CPU mode - and 0.05 seconds in CUDA mode "  ,how do you translator YOLO model from caffe to  init_net.pb and predict_net.pb in caffe2 ? Refer to your method above，i find  gpu mode costs much longer time than cpu mode ?  I don't know what the problem is，can you help me ?Thanks!

@ZIWEILI  can you use tiny-yolo model to test picture and recognize something in caffe2 ? I successfully recognize the content in caffe ,but fail to do it in caffe2. I find the data of detections = workspace.FetchBlob(out_name) are different from  out = net.forward_all(data=np.asarray([transformer.preprocess('data', img)])),  what is wrong ???

Hi diz-vara,
I tried your approach and ran into couple of issues. Any idea what might be the issue ?
Error1 :
Code
with open("predict_net.pb", 'rb') as f:
net_def.ParseFromString(f.read())
net_def.device_option.CopyFrom(device_opts)
workspace.CreateNet(net_def.SerializeToString())
Error
Required modules imported.
BATCH SIZE: 1
Config set!
Functions set.
Mean was set to: 128
INIT_NET = /local/caffe2/build/caffe2/python/models/bvlc_alexnet/init_net.pb
PREDICT_NET = /local/caffe2/build/caffe2/python/models/bvlc_alexnet/predict_net.pb
found /local/caffe2/build/caffe2/python/models/bvlc_alexnet/init_net.pb ...now looking for /local/caffe2/build/caffe2/python/models/bvlc_alexnet/predict_net.pb
all needed files found! loading the model in the next block
Number of batches: 1000
Pre-processing images...
Done pre-processing, initializing net...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1004 16:05:53.260538 10063 workspace.cc:157] Blob data not in the workspace.
Traceback (most recent call last):
File "alexnet_pretrain.py", line 202, in
workspace.CreateNet(net_def.SerializeToString())
File "/local/caffe2/build/caffe2/python/workspace.py", line 167, in CreateNet
StringifyProto(net), overwrite,
File "/local/caffe2/build/caffe2/python/workspace.py", line 190, in CallWithExceptionIntercept
print("Traceback for operator {} in network {}".format(op_id, net_name))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xb7 in position 164: ordinal not in range(128)
Error2
Code
When i changed the code to
with open("predict_net.pb", 'rb') as f:
net_def.ParseFromString(f.read())
net_def.device_option.CopyFrom(device_opts)
workspace.CreateNet(net_def)
Error
Required modules imported.
BATCH SIZE: 1
Config set!
Functions set.
Mean was set to: 128
INIT_NET = /local/caffe2/build/caffe2/python/models/bvlc_alexnet/init_net.pb
PREDICT_NET = /local/caffe2/build/caffe2/python/models/bvlc_alexnet/predict_net.pb
found /local/caffe2/build/caffe2/python/models/bvlc_alexnet/init_net.pb ...now looking for /local/caffe2/build/caffe2/python/models/bvlc_alexnet/predict_net.pb
all needed files found! loading the model in the next block
Number of batches: 1000
Pre-processing images...
Done pre-processing, initializing net...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1004 16:13:15.168926 17579 workspace.cc:157] Blob data not in the workspace.
Traceback for operator<PHONE_NUMBER> in network bvlc_alexnet
Traceback (most recent call last):
File "alexnet_pretrain.py", line 203, in
workspace.CreateNet(net_def)
File "/local/caffe2/build/caffe2/python/workspace.py", line 167, in CreateNet
StringifyProto(net), overwrite,
File "/local/caffe2/build/caffe2/python/workspace.py", line 186, in CallWithExceptionIntercept
return func(*args, **kwargs)
RuntimeError: [enforce fail at operator.cc:52] blob != nullptr. op Conv: Encountered a non-existing input blob: data

You need feed data blob to current workspace, or you can create ConstantFill operator and save it to init_net. Check https://zhuanlan.zhihu.com/p/45067948 for more detail.


Fix ICAControllerKeeper

Description
When I try to implement the ICA source code following the instructions I get this error message:
error while running command go build -o /Users/raul/go/bin/bcnad -mod readonly -ldflags -X github.com/cosmos/cosmos-sdk/version.Name=Bcna -X github.com/cosmos/cosmos-sdk/version.AppName=bcnad -X github.com/cosmos/cosmos-sdk/version.Version=1.5.0 -X github.com/cosmos/cosmos-sdk/version.Commit=a540ff758ec43d22fbf9cb78a97719ebd85c87ec -X github.com/BitCannaGlobal/bcna/cmd/bcnad/cmd.ChainID=bcna .: # github.com/BitCannaGlobal/bcna/app
../../app/app.go:404:3: too many arguments in call to icacontrollerkeeper.NewKeeper
        have (codec.Codec, *"github.com/cosmos/cosmos-sdk/store/types".KVStoreKey, "github.com/cosmos/cosmos-sdk/x/params/types".Subspace, "github.com/cosmos/ibc-go/v3/modules/core/04-channel/keeper".Keeper, "github.com/cosmos/ibc-go/v3/modules/core/04-channel/keeper".Keeper, *"github.com/cosmos/ibc-go/v3/modules/core/05-port/keeper".Keeper, "github.com/cosmos/cosmos-sdk/x/auth/keeper".AccountKeeper, "github.com/cosmos/cosmos-sdk/x/capability/keeper".ScopedKeeper, *baseapp.MsgServiceRouter)
        want (codec.BinaryCodec, "github.com/cosmos/cosmos-sdk/store/types".StoreKey, "github.com/cosmos/cosmos-sdk/x/params/types".Subspace, "github.com/cosmos/ibc-go/v3/modules/apps/27-interchain-accounts/types".ICS4Wrapper, "github.com/cosmos/ibc-go/v3/modules/apps/27-interchain-accounts/types".ChannelKeeper, "github.com/cosmos/ibc-go/v3/modules/apps/27-interchain-accounts/types".PortKeeper, "github.com/cosmos/cosmos-sdk/x/capability/keeper".ScopedKeeper, *baseapp.MsgServiceRouter)
: exit status 2

I fixed deleting app.AccountKeeper at the ICA Controller Keeper:
	// Create the Keeper for each submodule
	app.ICAControllerKeeper = icacontrollerkeeper.NewKeeper(
		appCodec, keys[icacontrollertypes.StoreKey],
		app.GetSubspace(icacontrollertypes.SubModuleName),
		app.IBCKeeper.ChannelKeeper, // may be replaced with middleware such as ics29 fee
		app.IBCKeeper.ChannelKeeper,
		&app.IBCKeeper.PortKeeper,
		scopedICAControllerKeeper,
		app.MsgServiceRouter(),
	)

closes: #XXXX

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

[ ] Targeted PR against correct branch (see CONTRIBUTING.md)
[ ] Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
[ ] Code follows the module structure standards.
[ ] Wrote unit and integration tests
[ ] Updated relevant documentation (docs/) or specification (x/<module>/spec/)
[ ] Added relevant godoc comments.
[ ] Added a relevant changelog entry to the Unreleased section in CHANGELOG.md
[ ] Re-reviewed Files changed in the Github PR explorer
[ ] Review Codecov Report in the comment section below once CI passes

Amazing team's response  😍


Fix memory leak in error case of GdipCreatePath

Fixes https://github.com/mono/libgdiplus/issues/224

Thanks! I started working on a more ambitious PR, but it fails test at the moment, so it may be easier to just fix the small things first.


html-xml-utils: 8.3 -> 8.4

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/html-xml-utils/versions.
meta.description for html-xml-utils is: "Utilities for manipulating HTML and XML files"
meta.homepage for html-xml-utils is: "http://www.w3.org/Tools/HTML-XML-utils/"
meta.changelog for html-xml-utils is: ""
Updates performed

Version update

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxremove had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxpipe had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxcopy had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxselect had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxaddid had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxmkbib had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxwls had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxunpipe had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxnsxml had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxref had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/asc2xml had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxxmlns had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxclean had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxextract had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/xml2asc had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxprune had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxincl had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxuncdata had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxcite had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxtabletrans had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxindex had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxcount had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxunxmlns had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxnormalize had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxtoc had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxprintlinks had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxname2id had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxmultitoc had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxcite-mkbib had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxnum had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin/hxunent had a zero exit code or showed the expected version
0 of 31 passed binary check by having a zero exit code.
0 of 31 passed binary check by having the new version present in output.
found 8.4 with grep in /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4
found 8.4 in filename of file in /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

1 package rebuild(s)

1 x86_64-linux rebuild(s)
1 x86_64-darwin rebuild(s)

First fifty rebuilds by attrpath
html-xml-utils

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A html-xml-utils https://github.com/r-ryantm/nixpkgs/archive/9d1895cd2418f87c8841170a118f005059d7339c.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4
ls -la /nix/store/6gnbbwphx2kk4rd57wxcscvwpavkwzsg-html-xml-utils-8.4/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

html-xml-utils

Maintainer pings

Result of nixpkgs-review pr 167219 run on x86_64-linux 1

1 package built:

html-xml-utils

Change looks okay.
* hxselect.c: The :attr() pseudo-element was only matched if it
was on the first selector, not if it was on a subsequent selector
in a comma-separated list of selectors. (Found and fixed by Bas
Ploeger.)

* tests/select33.sh: Added test for multiple :attr()
selectors. (Contributed by Bas Ploeger.)


Event Subscription does not work with Azure Communication Services

It always turns out to be the following error when trying to register an event subscription.   The error says, "event grid notifications are currently not supported by microsoft.communication in Global" Wondering how the given samples work as it's a global resource only we have a global location. Also,  Register / Re-Register does not help as well.
[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ab96c98a-2112-da55-dfa7-acd312870972
Version Independent ID: f1a499c5-eece-80d4-bccf-a0e5755f8d90
Content: Quickstart - Handle SMS events - An Azure Communication Services quickstart
Content Source: articles/communication-services/quickstarts/telephony-sms/handle-sms-events.md
Service: azure-communication-services
GitHub Login: @mikben
Microsoft Alias: mikben

@JanaAbhijit,  Thanks for the feedback! We are taking a look into this and will get back to you soon.

Thanks @AjayKumar-MSFT  , looking forward to a quick update on this, currently not able to move forward with basic communication testing.  Also, if I hit the endpoint, it always says services are not running

Abhijit, apologies for the delay! the above error typically means the REST call failed. Just to clarify, from your original post, was the registration successful?
We need additional details to identify the cause, so we wish to engage with you offline further for a closer look and provide a specialized assistance, could you kindly send an email  to AzCommunity[at]Microsoft[dot]com referencing this thread. I will follow-up with you.  Since we will work with you offline, we will close this thread for now.
Thanks for leveraging Azure docs feedback channel to raise this question! This channel is reserved for ‘Azure docs’ feedback- document feedback & content improvements.

Thanks, @AjayKumar-MSFT , I will drop an email.  I faced these issues with multiple subscriptions even where event grid services I have registered newly.

Abhijit,  thanks for sending the email and sharing additional details. I'm following-up with you further offline.

Abhijit, As we're following-up on this offline and also there is no specific doc enhancement at this time, we will close this thread for now.

Hey @AjayKumar-MSFT do we have the solution for the above? Can you please share it?

Abhijit, was able to resolve the issue after testing it again (on a different subscription), and it worked - was able to subscribe to some messages to queue from communication services.


Update dask to 2022.9.2

How would we go about updating dask to the 2022.9.2 version?
This version is supposed to fix a major problem that many run into with using dask: https://github.com/dask/distributed/discussions/7128#discussioncomment-3859002.
Do @jbusecke or @TomNicholas know how to do the update?

I believe there are bots that automatically handle updating packages. https://github.com/pangeo-data/pangeo-docker-images/pull/385 (opened 4 days ago) updates, among other things, dask + distributed to 2022.9.2. @scottyhq does CI passing on that PR mean the package updates are okay, or is there further testing that's usually done?

Big +1 from me. I would love to check out the 'worker-saturation' feature on the hubs too!

@jbusecke https://github.com/2i2c-org/infrastructure/pull/1769 is the equivalent on the hubs.

does CI passing on that PR mean the package updates are okay, or is there further testing that's usually done?

Yes. no further testing is done currently. But I do usually look over the PR diff to make sure there are no major package version drops and look at the actions log to make sure the image size doesn't change significantly. It would be awesome to add some automated checks for those things! and @yuvipanda i wonder if it's now a good time to revive the idea of automated testing of new images on the hub itself with a bot user (I know there has been discussion of this in the past)...

closed by #385


video cannot be played

Prerequisites

[ X] Put an X between the brackets on this line if you have done all of the following:

Checked that your issue isn't already filed: https://github.com/GetStream/Winds/issues

Description
[Description of the issue]
I cannot play video in the feed window.
Steps to Reproduce

[First Step]
[Second Step]
[and so on...]

Expected behavior: [What you expect to happen]
Actual behavior: [What actually happens]
Reproduces how often: [What percentage of the time does it reproduce?]
Versions
You can get this information from the manu. Winds > About Winds.
Additional Information
Any additional information, configuration or data that might be necessary to reproduce the issue.

@wcadaydayup Thank you for reporting this. Could you share the article link with this issue?


How can I write any message in any channel programatically instead of b.respond()?

How can I write any message in any channel programatically instead of b.respond()?
There are lots of examples about using b.respond but nothing about writing programatically instead of responding to something.

@online0227 - You can use the bot's dispatch method, to send an envelope, which can contain strings and/or attachment/s. The easiest way to compose the attributes of the envelope (other than passing them all into the constructor) is using its helpers...
const envelope = new bot.Envelope({ user: new bot.User() })
  .toRoomId('111')
  .write('hello')
bot.dispatch(envelope)

There aren't any examples I can find of this I could find easily, but the envelope.ts code is well commented so you can see the methods there. See here. I haven't version the docs generation, so the exact usage for your version might involve trial and error.
The rocketchat boilerplate also includes an example of using envelope methods to add quick reply buttons as attachments. See here.

@timkinnane thanks a lot for the information. I try hard to post a dm to a user but right now im stuck. can you shed some light how username has to be specified? tried it
new bot.Envelope( {user: {name: 'username'} })
and
new bot.Envelope( {user: {id: 'userid'} })
but I only get the follwing errors:
info: 	[thought] dispatch envelope f6d0b5763928a2baabfbbb37642c1d04 debug: 	[thought] global respond processing incoming message ID 3aba1b126cf5e9d6e04822d57c424d36 debug: 	[createDirectMessage] Calling (cached): undefined error: 	[createDirectMessage] Error:Match failed [400]
would be very happy about any help! thanks a lot for the great work!

to answer the question by my self:
const user = { username: 'da' }
bot.dispatch(new bot.Envelope({ user, method:'dm' }).compose('hello'))

ok, nevertheless: i can't get sending to channel by channel-name wo work.
The example from http://bbot.chat/docs/hubot
const room = { name: 'general' }
const user = { name: 'bilbo' }
bot.dispatch(new bot.Envelope({ room }).compose('hello #' + room.name))

throws an error complaining that no roomID has been provided in node_modules/bbot/dist/adapters/rocketchat.js line 172 and the send-case only calls this.driver.sendToRoomId and does not call this.driver.sendToRoom if only room.name but not room.id is provided.
If I implement the call to this.driver.sendToRoom accordingly its working fine.
@timkinnane can we discuss if this is correct? if it is i can do a pr...

@anicoa You're totally right. Seems the docs do not match the implementation. I'd be happy to accept a PR to fix it.
Seems like the condition below should be updated to throw if no room and no ID, otherwise if room name given without ID, use driver.sendToRoom.
https://github.com/Amazebot/bbot/blob/67c0088a807d3b57c2094f04f50e37314b034539/packages/bbot-message-rocketchat/src/rocketchat.ts#L112
I've been working on the next major release for a long time (without much time to progress it lately) and it's moved far away from master, so I would apply any changes as a patch to the 1.4.0 release and carry across your changes manually into my v2 dev branch.
If it helps, you could also chain the envelope utility methods to set the params you need, which might be more semantic. e.g. bot.dispatch(new bot.Envelope().toUser(user).via('dm').compose('hello')
Thanks for your efforts.


Security fix for Prototype Pollution

:bar_chart: Metadata *
@szydlovski/deep-object is vulnerable to Prototype Pollution.
Bounty URL: https://www.huntr.dev/bounties/1-npm-%40szydlovski%2Fdeep-object
:gear: Description *
Prototype Pollution refers to the ability to inject properties into existing JavaScript language construct prototypes, such as objects.
JavaScript allows all Object attributes to be altered, including their magical attributes such as __proto__, constructor and prototype. An attacker manipulates these attributes to overwrite, or pollute, a JavaScript application object prototype of the base object by injecting other values. Properties on the Object.prototype are then inherited by all the JavaScript objects through the prototype chain.
:computer: Technical Description *
Fix implemented by not allowing to modify object prototype.
:bug: Proof of Concept (PoC) *

Create the following PoC file:

// poc.js
var deepObject = require("@szydlovski/deep-object")
var obj = {}
console.log("Before : " + {}.polluted);
deepObject.setProperty(obj, '__proto__.polluted', 'Yes! Its Polluted');
console.log("After : " + {}.polluted);

Execute the following commands in terminal:

npm i @szydlovski/deep-object # Install affected module
node poc.js #  Run the PoC

Check the Output:

Before : undefined
After : Yes! Its Polluted

:fire: Proof of Fix (PoF) *

+1 User Acceptance Testing (UAT)

I've executed unit tests.
After fix the functionality is unaffected.

👋 Hello, @szydlovski - @arjunshibu has opened a PR to us with a fix for a potential vulnerability in your repository. To view the vulnerability, please refer to the bounty URL in the first comment, above.
Ultimately, you get to decide if the fix is 👍 or 👎. If you are happy with the fix, please write a new comment (@huntr-helper - LGTM) and we will open a PR to your repository with the fix. All remaining PRs for this vulnerability will be automatically closed.
If you have any questions or need support, come and join us on our community Discord!
@szydlovski & @arjunshibu - thank you for your efforts in securing the world’s open source code! 🎉

👋 Hello, @szydlovski - @arjunshibu has opened a PR to us with a fix for a potential vulnerability in your repository. To view the vulnerability, please refer to the bounty URL in the first comment, above.
Ultimately, you get to decide if the fix is 👍 or 👎. If you are happy with the fix, please write a new comment (@huntr-helper - LGTM) and we will open a PR to your repository with the fix. All remaining PRs for this vulnerability will be automatically closed.
If you have any questions or need support, come and join us on our community Discord!
@szydlovski & @arjunshibu - thank you for your efforts in securing the world’s open source code! 🎉


[WIP] Bug fixes in spike detection

New phy spikesort tool, replacing klusta

@nsteinme this is the PR I used to run phy spikesort on your data

rebased

Just to be clear, are you saying you'd like me to try it out now? If so,
what kind of command do I issue? Do I need to make a new prm file first?
On Mon, Jun 29, 2015 at 7:34 PM, Cyrille Rossant<EMAIL_ADDRESS>wrote:

@nsteinme https://github.com/nsteinme this is the PR I used to run phy
spikesort on your data
—
Reply to this email directly or view it on GitHub
https://github.com/kwikteam/phy/pull/448#issuecomment-116789380.

well, yes you can if you want, without any changes to the PRM for now, because I haven't implement the PRM changes yet.

ready for review/merge @nippoo

I changed KK2 so that now you don't pass an argument to cluster_mask_starts and the num_starting_clusters is a standard KK2 parameter like any of the others.

thanks @thesamovar! can you make a new pip release so that travis can pass here?

thanks @thesamovar! can you make a new pip release so that travis can pass here?

Done!

thanks! travis happy :)


Consider a license change

GPLv3 is a very restrictive license for a crate. LGPL license which is quite similar to GPL would be much better as a library as it wouldnt force any project that uses it to adopt GPLv3.

Agreed. I'd prefer to use the Apache 2.0 + MIT dual license typical for Rust crates, but much of the code was written very closely based on blackbox_decode and the blackbox encoding parts of BF, INAV, etc which are all GPLv3. The GPL faq is pretty clear that ports/translations need to stay under the same license. Perhaps the overall algorithm I use is different enough from blackbox_decode that it is novel work not simply a port but that is not a distinction I feel qualified to make.
Perhaps @thenickdude who wrote much/most of blackbox_decode could weigh in?

This looks like substantial enough of a rewrite that I doubt the original license from my code applies any more.
The main licensing concern in blackbox_decode was that it included a copy/paste of imu.c from Baseflight which itself derived from GPLv3 MultiWii, but that doesn't seem to be present here?

@wetheredge aside from bot commits this is the only commit you did not write. https://github.com/blackbox-log/blackbox-log/commit/48d5271d3e97892f6d5fc6982f4e168e6fc588c6
And I would agree with thenickdude that you wouldnt be forced to use GPLv3 as this is enough of a new work rather than just a derivative work.

I'll publish v0.4.0 with the new license soon.


Stop independent services in kube-compose defined in docker-compose.yml

kube-compose down should be able to stop independent services.
For eg:  kube-compose down service-1

K8s services should only be deleted if down would stop all compose services. This is so that existing pods will not have their hostAliases invalidated by a partial down. For example, suppose we have a docker-compose file with two services a and b, and we run the following commands:
kube-compose up
kube-compose down a
kube-compose up a

If kube-compose down a deletes the k8s service of a, then the host aliases set on the pod for b are no longer valid (because the k8s service of a has a new IP).


not running stdlib.h

a.TXT
THE FILE RENAMED TO a.TXT
a.c(1): fatal error C1083: Cannot open include file: 'stdlib.h': No such file or
directory
i want to run system calls linux (learning OS course BSCS )
but it isn't compiling
from:
https://docs.microsoft.com/en-us/cpp/build/walkthrough-compile-a-c-program-on-the-command-line?view=vs-2019
what are suggestions so that i can run code on linux but debug on visual studio

@muhammadfawad1
It looks like you're on a page with an example meant for Windows. For Linux, I'd start with Download, install, and set up the Linux workload to make sure you've got everything set up correctly. The Windows Subsystem for Linux (WSL) is a convenient way to try things out, although you can also target another Linux computer.
Once that's set up, you need to Connect to your target Linux system in Visual Studio. There are separate instructions for a remote box or for WSL connections.
Next, follow the steps in Create a new Linux project, and choose the Console Application (Linux) template. It starts with a "Hello, World" app that you should be able to edit into what you really want to build. At the bottom of the article, follow the Next steps links to move on to articles about configuring, deploying, and debugging your programs.
I'm closing your issue, since it doesn't seem to show a problem with this particular article. Good luck with your schoolwork.


Add NSUbiquitousKeyValueStore.url(forKey:) solving issue #910

Checklist

[x] I checked the Contributing Guidelines before creating this request.
[x] New extensions are written in Swift 5.0.
[x] New extensions support iOS 10.0+ / tvOS 9.0+ / macOS 10.10+ / watchOS 2.0+, or use @available if not.
[x] I have added tests for new extensions, and they passed.
[x] All extensions have a clear comments explaining their functionality, all parameters and return type in English.
[x] All extensions are declared as public.
[x] I have added a changelog entry describing my changes.

          1 Message

:book:
Thank you for submitting a pull request to SwifterSwift. The team will review your submission as soon as possible.

  Generated by :no_entry_sign: Danger

Closing due inactivity, feel free to reopen if you still want to move this forward :)


New Tree structure

Tree Hashing:

Leaf Node: sha3
Internal Node: blake2b
Modify tree.NewDefaultTree() with correspondent options.

Account Sign:
func (acc *Account) SignMsg(key KeyType, msg []byte) (*coredocumentpb.Signature, error) {
switch(key):
case "secp":
    return signSecp(msg)
case "jub": // Not implemented yet
    return signJub(msg)
}

Request Signatures:
func RequestDocumentSignature() {
    ...
    ethDR := model.GetEthDataRoot()
    zkDR := model.GetZKDataRoot()

    sigs := make([]*coredocumentpb.Signature, 2)
    sigs[0] := acc.SignMsg(key.Secp, ethDR)
    sigs[1] := acc.SignMsg(key.Jub, zkDR)

    model.AppendSignatures(sigs...)
    ...
}

Validate Signatures:
func ValidateSignature(did, docRoot, sig) error {
    flag := sig.TransitionValidated
    payload := DocConsensusSignature(docRoot, flag)
    // Adding Type of signature
    return idService.ValidateSignature(did, sig.Type, sig.Pk, payload)
}

Extend the proto msg to include the curve type, excluding from tree, only as p2p message.
Or have a try-error approach on the validator invoker that at least with one key needs to be validated.

By #1293


Projects missing .objectTranslation when having .fieldTranslation fail on CLI/VSCode operations

Summary
Projects created by earlier versions of the CLI do not create an empty .objectTranslation file even when there is a .fieldTranslation file. This results in modern versions of the CLI throwing an "Unexpected child metadata" error on any operation, even those not including that metadata. This means the developer is unable to do various actions, such as retrieving metadata using the org browser via VS Code. Note this includes operations that don't touch the CustomObjectTranslation.
My suggestion is twofold:

Make this particular metadata type more resilient against this error
Make it so less operations perform a scan that can throw this error.  If it does, it should warn rather than error.

Example error:
Error (1): Unexpected child metadata [/force-app/main/default/objectTranslations/Asset-en_US/asset_address.fieldTranslation-meta.xml] found for parent type [CustomObjectTranslation]
Steps To Reproduce
Creating a broken project using sfdx:

Create fresh SF instance
Install an older version of sfdx, I used npm to install a project specific instance of the command.
On an object without an object translation, add a field translation. I used Asset.Address for my tests
Create a package.xml file including CustomObjectTranslation
Run the retrieve using the older version of sfdx. This is what I ran: npx sfdx force:source:retrieve -x package.xml --verbose
Note that within "objectTranslations" you will see an Asset folder with only a .fieldTranslation file

Creating a broken project using updated commands:

Run steps 1,3,4
Run the retrieve using the up to date sf command sf project retrieve start --manifest package.xml
Within objectTranslations/Asset-en_US you will see both an .objectTranslation and .fieldTranslation. Delete the .objectTranslation

Getting error with the broken project:

Run: sf project retrieve start --metadata ApexClass:ExampleClass
Go into org browser and try to retrieve any component

Expected result
Command should complete without errors. Throwing warnings make more sense in this context.
Actual result
Command throws error:
Error (1): Unexpected child metadata [/force-app/main/default/objectTranslations/Asset-en_US/asset_address.fieldTranslation-meta.xml] found for parent type [CustomObjectTranslation]
System Information
{
  "architecture": "darwin-arm64",
  "cliVersion"<EMAIL_ADDRESS>  "nodeVersion": "node-v20.11.1",
  "osVersion": "Darwin 23.4.0",
  "rootPath": "/Users/<user>/.local/share/sf/client/2.35.6-3a4a215",
  "shell": "zsh",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.0.13 (core)",
    "@oclif/plugin-commands 3.2.2 (core)",
    "@oclif/plugin-help 6.0.20 (core)",
    "@oclif/plugin-not-found 3.1.1 (core)",
    "@oclif/plugin-plugins 5.0.1 (core)",
    "@oclif/plugin-search 1.0.20 (core)",
    "@oclif/plugin-update 4.2.4 (user)",
    "@oclif/plugin-version 2.0.16 (core)",
    "@oclif/plugin-warn-if-update-available 3.0.15 (core)",
    "@oclif/plugin-which 3.1.7 (core)",
    "@salesforce/cli 2.35.6 (core)",
    "apex 3.1.0 (core)",
    "auth 3.5.0 (core)",
    "data 3.2.2 (core)",
    "deploy-retrieve 3.4.0 (core)",
    "info 3.1.0 (core)",
    "limits 3.2.0 (core)",
    "marketplace 1.1.0 (core)",
    "org 3.6.0 (core)",
    "packaging 1.26.3 (user)",
    "schema 3.2.0 (core)",
    "settings 2.1.0 (core)",
    "sobject 1.2.0 (core)",
    "source 3.2.0 (core)",
    "telemetry 3.1.17 (core)",
    "templates 56.1.0 (core)",
    "trust 3.4.0 (core)",
    "user 3.4.0 (core)"
  ]
}

Additional information
Related stories:

Similar issue, different metadata types: https://github.com/forcedotcom/cli/issues/2804
The same issue (as far as I can tell) appears to have been fixed in sfdx previously: https://github.com/forcedotcom/cli/issues/1241
Project file: ObjTranslationIssues.zip

Doctor debug.log<PHONE_NUMBER>871-command-debug.log
Stack trace from doctor debug log:
TypeInferenceError: Unexpected child metadata [<projectPath>/force-app/main/default/objectTranslations/Asset-en_US/asset_address.fieldTranslation-meta.xml] found for parent type [CustomObjectTranslation]
    at DecomposedSourceAdapter.populate (/Users/<user>/.local/share/sf/client/2.35.6-3a4a215/node_modules/@salesforce/source-deploy-retrieve/lib/src/resolve/adapters/decomposedSourceAdapter.js:104:23)
    at DecomposedSourceAdapter.getComponent (/Users/<user>/.local/share/sf/client/2.35.6-3a4a215/node_modules/@salesforce/source-deploy-retrieve/lib/src/resolve/adapters/decomposedSourceAdapter.js:69:21)
    at MetadataResolver.resolveComponent (/Users/<user>/.local/share/sf/client/2.35.6-3a4a215/node_modules/@salesforce/source-deploy-retrieve/lib/src/resolve/metadataResolver.js:125:44)
    at MetadataResolver.getComponentsFromPathRecursive (/Users/<user>/.local/share/sf/client/2.35.6-3a4a215/node_modules/@salesforce/source-deploy-retrieve/lib/src/resolve/metadataResolver.js:85:40)
    at /Users/<user>/.local/share/sf/client/2.35.6-3a4a215/node_modules/@salesforce/source-deploy-retrieve/lib/src/resolve/metadataResolver.js:107:63
    at Array.flatMap (<anonymous>)
    at MetadataResolver.getComponentsFromPathRecursive (/Users/<user>/.local/share/sf/client/2.35.6-3a4a215/node_modules/@salesforce/source-deploy-retrieve/lib/src/resolve/metadataResolver.js:107:43)
    at /Users/<user>/.local/share/sf/client/2.35.6-3a4a215/node_modules/@salesforce/source-deploy-retrieve/lib/src/resolve/metadataResolver.js:107:63
    at Array.flatMap (<anonymous>)
    at MetadataResolver.getComponentsFromPathRecursive (/Users/<user>/.local/share/sf/client/2.35.6-3a4a215/node_modules/@salesforce/source-deploy-retrieve/lib/src/resolve/metadataResolver.js:107:43)

Workaround involves deleting objectTranslations folder (works as long as you don't need to commit changes against it). This will cause git to throw a fit, so here is the workaround I found. Note you have to run this whenever changing branches:

cd into objectTranslations folder, run: find . -maxdepth 1 -type d \( ! -name . \) -exec bash -c "cd '{}' && pwd && git ls-files -z ${pwd} | xargs -0 git update-index --skip-worktree" \;
Delete objectTranslations

Hey @evan-cm - how old was the "old" version of the CLI you were using? Over the years we've had to adjust, fix, or redo how metadata was represented locally, and sometimes it's not always backwards compatible. During the transitions we've always warned and given appropriate time to adjust or prepare for a difference. If you update to the latest, as of last week, you can customize how metadata is represented locally using our Custom Decomposition.

@WillieRuemmele It appears the old CLI version is 7.119.2 ~3 years old. While I'm not sure the Custom Decomposition is helpful here, I appreciate you calling it out as it looks like a neat feature that I'm interested in playing with.
While I understand metadata format may change (I don't believe this is exactly what happened here), I believe my two suggestions still apply as they expand beyond this issue and helps address the general class of errors this error falls into. A project shouldn't be completely unusable because one component somewhere is malformed. Why does apex code manipulations break when an unrelated component is malformed? Warning the user on unrelated operations, and breaking on operations on that metadata makes sense to me. This would help alleviate these sorts of issues: https://github.com/forcedotcom/cli/issues?q=is%3Aissue+Unexpected+child+metadata. I also wonder about the efficiency of validating the entire package on every command, although I haven't noticed performance issues so maybe that doesn't matter.
On top of that, the error is a little confusing for the user. It feels weird that the tooling cannot infer the parent metadata from the folder name seeing as the objectTranslation file shares the name as the containing folder.
To reiterate, my two suggestions for fixes are:

Update what I presume to be a project validation scan at the beginning to only warn the user if errors are found on unrelated metadata types
Re-evaluate the logic for finding the parent component, do you need the objectTranslation folder, or can you get the required information from the containing folder name. Note I'd suggest a similar fix for other components that have the same sort of issues like this one.

As an aside, if anyone else is wanting to mass generate the blank objectTranslation files, this is a shell script to do so. Note that I'd still validate and ensure deploying the blank objectTranslation file does what you expect:
for brokenFile in $(diff  <(for x in force-app/main/default/objectTranslations/*/*.objectTranslation-meta.xml; do echo "$(dirname "$x")" ; done) <(printf "%s\n" force-app/main/default/objectTranslations/*) | grep ">" | sed 's!> force-app/main/default/objectTranslations/!!' )
do
    echo "Creating force-app/main/default/objectTranslations/$brokenFile/$brokenFile.objectTranslation-meta.xml"
    printf "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<CustomObjectTranslation></CustomObjectTranslation>\n" > force-app/main/default/objectTranslations/$brokenFile/$brokenFile.objectTranslation-meta.xml
    git add force-app/main/default/objectTranslations/$brokenFile/$brokenFile.objectTranslation-meta.xml
done

@evan-cm I've been thinking through your ideas.

Update what I presume to be a project validation scan at the beginning to only warn the user if errors are found on unrelated metadata types

it's not exactly what you're thinking (that we're doing a scan for project validation purposes).  There are a few reasons we  "resolve" stuff in the project before a retrieve

conflict detection.  If there are local changes in a file, and remote changes that map to those, we'd tell you you have conflicts and that you'd need to --ignore-conflicts so remote or local can "win".
merging.  We need the graph of the project so that when the retrieve happens, we can ask the question, "where do I put this?"  For example, if you have an apex class called Foo in a non-default packageDir, and we retrieve a class called Foo, we need to recognize that those match and that we are updating that local Foo instead of creating a new one in the default dir.

I see your point, that for both 1 and 2, we could be more selective about what we scan (ex: we figure out what's in your retrieve flags and THEN use that information to resolve a only a subset of the local project).

As far as "why is the parent file COT required, anyway" ?  Here's the metadata API docs
https://developer.salesforce.com/docs/atlas.en-us.api_meta.meta/api_meta/meta_'.htm
COFT are children of COT to the metadata API.  It wants a file for the COT, and all the COFT are an array of props inside that file.
It's safer/cleaner/more deterministic to roll up the COFT files into the COT and deploy that as whole.  If we were to create a COT file empty, and then add only your COFT, I'm not sure how the API interprets that.  Does that mean "only update what is different (the COFT)?"  or does it mean, "clear the description and gender properties from the COT? since they're not in the file anymore"   [if your COT doesn't have those, and really is empty, that's one thing.  But creating empty files where what's on the server might not be empty is sketchier.  You'd be better off re-retrieving the COT than generating blank files via a script]

Anyone else seeing breaking changes for CO(F)T?


Please release packages targeting netstandard2.0

Blocking #901

Here are sample packages: https://github.com/bherila/Swashbuckle.AspNetCore/releases/tag/4.0.0

The master branch in this repo already targets netstandard2.0. The official nuget release is still pending another feature but you can download packages that are built from latest master from myget.org. So sample packages are available

you can download packages that are built from latest master from myget.org

This is helpful, thank you!

No worries. Although it does look like they’re changing things up a bit, possibly pushing for more paying users, so finding the download page may be difficult. Use the following list no and you should be good:
https://www.myget.org/feed/domaindrivendev/package/nuget/Swashbuckle.AspNetCore


Update data domiciling docs

Fixes:

DOC-7796
DOC-7812
DOC-7756

Summary of changes:

Update 'Data Domiciling with CockroachDB' as follows:

Add a (default) option to use the ALTER DATABASE ... ADD SUPER REGION statement

Retain the (NON-default) instructions to use ALTER DATABASE ... PLACEMENT RESTRICTED in part because those instructions also work and are already vetted.  Also, having more ways to do a thing is usually good.

Update the instructions on checking for noncompliant replicas to use the HTTP-based 'critical nodes status endpoint' instead of the now-very-deprecated and non-working SQL replication reports.

Also, update the example cluster setup to be a secure cluster, such that using the critical nodes status endpoint shows an example of how to get an authentication cookie (of which there is currently only one in our entire doc set, unless I'm missing something).

Can we also leave a note about users can't store PII in columns that are part of an index if they want data to not be domiciled in incorrect regions.

AIUI we have such a note, it's the first one in the 'Limitations' section (if I'm understanding you correctly, please let me know if not):

When columns are [indexed]({% link {{ page.version.version }}/indexes.md %}), a subset of data from the indexed columns may appear in [meta ranges]({% link {{ page.version.version }}/architecture/distribution-layer.md %}#meta-ranges) or other system tables. CockroachDB synchronizes these system ranges and system tables across nodes. This synchronization does not respect any multi-region settings applied via either the [multi-region SQL statements]({% link {{ page.version.version }}/multiregion-overview.md %}), or the low-level [zone configs]({% link {{ page.version.version }}/configure-replication-zones.md %}) mechanism.

@arulajmani can we please get a review on this? It's been open for more than a month and I'd like to give our users the benefit of these updates.

FYI the contents of this PR have now been approved by Legal.  I will now take the following steps:

update the PR so it will merge
port the changes to the v23.2 docs as well
merge it


Single file upload - etag verification fails

Confirm by changing [ ] to [x] below:

[x ] I've gone though Developer Guide and API reference
[ x] I've searched for previous similar issues and didn't find any solution

Platform/OS/Hardware/Device
What are you running the sdk on?
Ubuntu 18.04
Describe the question
I am doing a single file upload of 1mb, it successfully uploads with the contentHash I set. I verified the contents of the file are correct.
However, when i verify the etag with content hash on putobject response, content hash in the outcome and etag does not match.
Am I doing this correctly?
/*
* TODO: verification from our side.
* S3 verifies if content hash is set during putobject request.
*
Aws::Utils::ByteBuffer digest = Aws::Utils::HashingUtils::CalculateMD5(*request.GetBo  dy());
Aws::String contentHash = Aws::Utils::HashingUtils::Base64Encode(digest);
if (outcome.GetResult().GetETag() == contentHash) {
cout << "Verified uploaded content" << handle->GetKey();
} else {
cout << "Verificaton failed" << contentHash << outcome.GetRe  sult().GetETag() << handle->GetKey();
      }

*/
Logs/output
If applicable, add logs or error output.
To enable logging, set the following system properties:
REMEMBER TO SANITIZE YOUR PERSONAL INFO
options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;
Aws::InitAPI(options)

Hi @kkbachu ,
Haven't tried this but just as a quick look from our tests the way we verify etags is with HexEncode can you try that?
Aws::StringStream ss;
ss << "\"" << HashingUtils::HexEncode(HashingUtils::CalculateMD5(*putObjectRequest.GetBody())) << "\"";
ss.str().c_str() == putObjectOutcome.GetResult().GetETag().c_str())

Thank you . Above suggestion worked.


porcupine categorize

Thank you for your excellent work! I have a question for you, why are porcupines categorized in primates? Is there a clerical error here?

Thank you for your questions. Before the final submission, to make the table 2 more readable, we added common names and adjusted the order of the list so that errors occurred. Porcupine is actually a rodentia.
刘丹
@.***
签名由 网易灵犀办公 定制
Original:
From：xpx9826 @.>Date：2023-11-01 09:52:22(中国 @.>Cc：Subscribed @.>Subject：[LoTE-Animal/LoTE-Animal.github.io] porcupine categorize (Issue #1)
Thank you for your excellent work! I have a question for you, why are porcupines categorized in primates? Is there a clerical error here?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: @.>

Thank you for your response. I have another question, you mentioned in your paper that the dataset has a total of 28K images, but I downloaded it and found that it is only 22K, does the 28K include the network images?

Yes, 28K include the network images
刘丹
@.***
签名由 网易灵犀办公 定制
Original:
From：xpx9826 @.>Date：2023-11-12 16:37:33(中国 @.>Cc：LoTE-Animal @.> , Comment @.>Subject：Re: [LoTE-Animal/LoTE-Animal.github.io] porcupine categorize (Issue #1)
Thank you for your response. I have another question, you mentioned in your paper that the dataset has a total of 28K images, but I downloaded it and found that it is only 22K, does the 28K include the network images?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: @.***>


Stress test

Stress test:

produces configurable number of log files and lines (limited by ulimit)
acts as ingester endpoint for agent (<IP_ADDRESS>:7080)
checks for duplicate and lost lines
prints periodic reports until all lines received from agent
supports all agent versions
supports gzip compression
does not override existing log files, appends by default
todo: add https support

Running from shell

1.1 start agent with env vars:
export LOGDNA_LOG_DIRS=/home/dmitri/SOURCE/TMP/root/test
export LOGDNA_HOST=localhost:7080
export LOGDNA_USE_SSL=false

1.2 start test
$ python utils/stress_test/stress_test.py /home/dmitri/SOURCE/TMP/root/test 10 5000
2023-10-19 22:35:27 INFO   starting ingestor web server
 * Serving Flask app 'stress_test'
 * Debug mode: off
2023-10-19 22:35:33 INFO
2023-10-19 22:35:33 INFO   total seq:           10
2023-10-19 22:35:33 INFO   received seq:        10
2023-10-19 22:35:33 INFO   completed seq:       10
2023-10-19 22:35:33 INFO   total lines:         50000
2023-10-19 22:35:33 INFO   committed lines:     50000
2023-10-19 22:35:33 INFO   received lines:      50000
2023-10-19 22:35:33 INFO   duplicate lines:     0
2023-10-19 22:35:33 INFO   unrecognized lines:  0
2023-10-19 22:35:33 INFO   committed line rate: 8341 per sec
2023-10-19 22:35:33 INFO   received line rate:  8643 per sec
2023-10-19 22:35:33 INFO   received line bytes: 3939 KB
2023-10-19 22:35:33 INFO   run time:            6 sec
2023-10-19 22:35:33 INFO   FINISHED in 6 sec

Running in Docker

2.1 build test image
cd utils/stress-test
docker build -t  logdna-agent-stress-test

2.2 run test and agent images
docker network create stress_test
docker run --rm --net stress_test --name stress-test -v $(pwd)/logs:/var/log logdna-agent-stress-test:latest /var/log 50 1000000
docker run -it --net stress_test -e MZ_INGESTION_KEY=blah -e LOGDNA_LOG_DIRS=/var/log -e LOGDNA_HOST=stress-test:7080 -e LOGDNA_USE_SSL=false -it -v $(pwd)/logs:/var/log logdna/logdna-agent:3.9.0-dev

Help

$ python utils/stress_test/stress_test.py -h
usage: stress_test.py [-h] [--line_rate LINE_RATE] [--override] log_dir num_log_files num_lines

Agent Stress Test

positional arguments:
  log_dir               Directory where log files are stored.
  num_log_files         Number of log files to use.
  num_lines             Number of lines to add to each log file.

optional arguments:
  -h, --help            show this help message and exit
  --line_rate LINE_RATE
                        Line rate per log file. (default: 100)
  --override            Override existing log files. (default: False)

I'm testing this in docker and it seems to be very slow:
docker network create stress_test
docker run --rm --net stress_test --name stress-test -v $(pwd)/logs:/var/log logdna-agent-stress-test:latest /var/log 50 1000000
docker run -it --net stress_test -e MZ_INGESTION_KEY=blah -e LOGDNA_LOG_DIRS=/var/log -e LOGDNA_HOST=stress-test:7080 -e LOGDNA_USE_SSL=false -it -v $(pwd)/logs:/var/log logdna/logdna-agent:3.9.0-dev

2023-10-19 13:57:12 INFO   total seq:           50
2023-10-19 13:57:12 INFO   received seq:        50
2023-10-19 13:57:12 INFO   completed seq:       0
2023-10-19 13:57:12 INFO   total lines:         50000000
2023-10-19 13:57:12 INFO   received lines:      30074
2023-10-19 13:57:12 INFO   duplicate lines:     0
2023-10-19 13:57:12 INFO   unrecognized lines:  0
2023-10-19 13:57:12 INFO   line rate:           497 per sec
2023-10-19 13:57:12 INFO   run time:            60 sec

limited by line creation rate - by delay in the writer loop https://github.com/logdna/logdna-agent-v2/pull/572/files#diff-a3b59d908d4e8b6e9d752b3f48add84059bc02c16e1aaec1a99a39559959ca50R172
50 * 10 lines p/s  * 50 = 500
i was thinking number of files should scale it.  parameterize the delay as line rate per file?

ahh, I missed that.  Making that configurable sounds like a good idea because you may not want to keep scaling out the number of files.

@teriyakichild
ready for review
number of improvements:

line_rate option, override_files option, README
added committed line reports from writers, committed line rate
added received line bytes, fixed committed lines in report

@dkhokhlov this looks good.    Are we encroaching on the performance limits of the agent?

@dkhokhlov this looks good. I do see these errors in the logdna-agent when running the stress-test with 30 or more log files:
2023-10-20T13:32:26.607403Z  WARN notify_stream: unexpected event, kind: Other, from_path: None, to_path: None, attrs: EventAttributes { inner: Some(EventAttributesInner { tracker: None, flag: Some(Rescan), info: None, source: None, process_id: None }) }

Is this expected? Are we encroaching on the performance limits of the agent?

This warning means that 3rd party component that provides file notifications to agent has experienced an overrun.  I saw this msg too under stress load close to the end of the test. Amazingly it did not affect the test. This is a subject for further testing. We did not see this msg before. Means stress test works. :)

okay cool, that makes sense.  I see some of the checks are failing so I will approve once those are fixed.


⚠ Bofrost.at has degraded performance

In 185e4c7, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1270 ms

Resolved: Bofrost.at performance has improved in fbcd7eb after 6 minutes.


Update vcpkg baseline to the October 2022 tag.

Looks like all the linux legs are failing because the vcpkg version installed is different than the ones we need. I am wondering if it is a vcpkg caching issue, or possibly the vcpkg version pre-installed on the linux machines are out-dated and need to be fixed.
-- Running vcpkg install
Error: while checking out baseline from commit '94ce0dab56f4d8ba6bd631ba59ed682b02d45c46' at subpath 'versions/baseline.json':
error: git failed with exit code: (128).
fatal: path 'versions/baseline.json' exists on disk, but not in '94ce0dab56f4d8ba6bd631ba59ed682b02d45c46'

This may be fixed by updating vcpkg to the latest master via `git pull` or fetching commits via `git fetch`.

You can use the current commit as a baseline, which is:
    "builtin-baseline": "ce99c947b64ea3c0e274e18cc3035ca727681089"
-- Running vcpkg install - failed
CMake Error at /usr/local/share/vcpkg/scripts/buildsystems/vcpkg.cmake:868 (message):
  vcpkg install failed.  See logs for more information:
  /mnt/vss/_work/1/s/build/vcpkg-manifest-install.log
Call Stack (most recent call first):
  /usr/local/share/cmake-3.24/Modules/CMakeDetermineSystem.cmake:124 (include)
  CMakeLists.txt:40 (project)

cc @danieljurek, @weshaggard, @BillyONeal

As long as the E2E tests pass and it compiles looks okay to me.

@ahsonkhan I'm not an expert but based on the issue https://github.com/microsoft/vcpkg/issues/23748 it looks like it might have to do something with the builtin-baseline sha.

Error: while checking out baseline from commit '94ce0dab56f4d8ba6bd631ba59ed682b02d45c46' at subpath 'versions/baseline.json':
error: git failed with exit code: (128).
fatal: path 'versions/baseline.json' exists on disk, but not in '94ce0dab56f4d8ba6bd631ba59ed682b02d45c46'

This probably means you're missing a fetch? https://github.com/microsoft/vcpkg/commit/94ce0dab56f4d8ba6bd631ba59ed682b02d45c46 exists.

@BillyONeal, I can try adding a vcpkg fetch step in our CI pipelines/machines, but I am wondering why this would only fail on linux. Does linux machines in AzDo come with an existing clone of vcpkg (or vcpkg pre-bootstrapped/installed)?

Otherwise, why would a fetch be required on a clean clone of a repo, to checkout an older commit sha?

I'm not sure, is this actually a clone? Unfortunately I don't know about how you invoke us and it isn't obvious from this change. Doing a which vcpkg may help check that we're getting the right copy?
I do observe that the commit you're explicitly trying to checkout is older than the baseline you're trying to use, so there may be versions referenced by that baseline that are unreachable by the version database in use?

I do observe that the commit you're explicitly trying to checkout is older than the baseline you're trying to use

Really?
The commit sha I am checking out should be identical to the baseline. I am using the released tag:
94ce0dab56f4d8ba6bd631ba59ed682b02d45c46 (tags/2022.10.19)
https://github.com/microsoft/vcpkg/releases/tag/2022.10.19

/azp run c - client - tests

/azp run c - client - tests

Fallout issues discovered from this work:
https://github.com/Azure/azure-sdk-for-c/issues/2403
https://github.com/Azure/azure-sdk-for-c/issues/2404

Based on recommendation from Billy, we decided to avoid using VCPKG_ROOT or a "system-level" vcpkg repo, and instead use a repo-specific/local copy of vcpkg. This way we don't rely on the pre-installed version that happens to come with the build machines.


Add information page when you visit the secret URL

This should contain

Views left
Time left to view the secret
Other information if available

Hello!
I am new to contributing to open-source projects and I want to resolve this issue. Can you please assign me this issue? Thank you :)

Sure. Go for it, and thanks! @blazenn2
Feel free to do the ux youd like for this, as long as it is easy to read. Library used is mantine.dev.

Hey! @bjarneo
can you elaborate more on how you want this feature to be implemented.
since on secret URL views left is already there, time left is not present so do you want the time left to be shown same as view left or create button like some thing that will show the information or lead to separate page.
and also will it be better if views left decrease by 1 live when someone opens the secret.

Hi,
I think I will close this issue as it is really not needed. Thank you for participating 🥇


Do not create any producer if the output type of a function is void

Motivation
If the function output type is void and an output topic is specified, currently we create a producer thats wasteful(since nothing is going to be produced anyways). This pr avoids the creation in that scenario
Modifications
Describe the modifications you've done.
Result
After your change, what will change.

@srkukarni I think you also need to check if the function result is null, don't call write on the sink

@jerrypeng added the check during write.

@jerrypeng reverted the check in write

run integration tests


VSCode ext: Custom tasks prefixed with .: instead of ccd: or concordium:

Bug Description
The configurable tasks are prefixed with .: instead of ccd: or concordium:.
In comparison, the regular rust tasks are prefixed with rust:.
I assume that this can lead to confusion, if you also have tasks for smart contracts on other chains. For example for the build task, which has the full name .: Build smart contract
Steps to Reproduce

Run Tasks: Configure Task
See the names of the different tasks available

Expected Result
rust: cargo clean
...
concordium: Build smart contract
concordium: Test smart contract
Actual Result
rust: cargo clean
...
.: Build smart contract
.: Test smart contract
Versions

Software Version: 0.1.0-0
OS: macOS Ventura (Apple M2)

Right, will fix this.
Just for information, it only happens when the smart contract project root is opened in VS Code, as the . is currently representing the relative path where the task originated from.
So when opening a directory with multiple smart contracts, this becomes the path to each of them, so they can be distinguished.


Issue36 spm

Merge usage of the Swift Package Manager to the default branch.

Looks good to me @sjjhsjjh . Ive tweaked the build phases tab of the target "App" - so that nobody has to keep changing the build version. This should make it easier when we migrate to Travis CI (@jcope )

Hi, I had a chat with Will and backed out just the changes to the shellscript because they don't pertain to Swift Package Manager.


Remove Increase Limit On Widget

There should not be a limit when increasing the value on a widget.

#4

Problem resolved with pull request #4

Thanks


Mobile setup

I added to this portfolio the following
index.html
style.css
.hintrc
.gitignore

@Olamarx i have tried to implement the background through the css but its not working what should I do please

@Olamarx i have tried to implement the background through the css but its not working what should I do please

It will work, just try to read this post and follow it.


Annotations Processor

Hi,
JFYI: I started a new project https://github.com/stokito/jpatterns-annotations-processor
My idea is to merge this project http://www.dp4j.com/ that already have processors of Singleton and template Method. But it has GNU license and I'll ask author to import the project to GitHub and republish it under Apache v2 licence.
If anybody wants to contribute - you welcome.

Hi Stokito, did anything happen with your JPatterns Annotations Processor?

Hi @kabutz unfortunately no, I didn't finished a work. If you interested we can continue work on project and release it.

It's nice to see people are still interested!

You'll be really surprised how often I still sell my Design Patterns Course (I certainly am).  We even did one last week in Paris.

Hello,
I created an annotation processor that checks some properties of annotated classes, methods and fields. If it thinks there is a bug in design pattern implementation, it reports a compilation warning (or error, or nothing - it can be configured). The compilation message can be produced for example if ConcreteDecorator class is abstract, or if it and it's superclasses don't contain any field of Component type.
It is currently in my fork of JPatterns: https://github.com/aczaplinski/jpatterns . This fork also merges all important changes from all forks of JPatterns here, upgrades Java to version 11 and fixes some bugs.
You can see the annotation processor in action in my Mockito fork: https://github.com/aczaplinski/mockito and my Guava fork: https://github.com/aczaplinski/guava . You have to provide the compiled version of my JPatterns fork for them to work.
What do you think of it?


SSL SYSCALL EOF detected

Describe the bug
cross posted from https://github.com/sqlalchemy/sqlalchemy/issues/5955
I am really sorry if this is not the right place for this, but I am trying to cover all of my bases, throw out everything I have and see if I get a bite anywhere. Any help you may have to offer is greatly appreciated!
Hello, we are plagued by the never ending SSL SYSCALL error. We are running out postgres database on an RDS using a db.t2.large. Resource wise it has plenty of space and resources, and the connections are set to an amount that is no where near being hit. We have been plagued with this issue for a long time, but I just recently was assigned to the issue. This tends to happen when we the following uwsgi ini:
[uwsgi]
binary-path = /usr/sbin/uwsgi
virtualenv = /opt/webapps/iamp/venv
strict = false
module = wsgi
plugins = python36
buffer-size = 16384
die-on-term = true
lazy-apps = false #make this true to fix syscall issue
single-interpreter = true

# NETWORK CONFIG
socket = :8080
chmod-socket = 660

# WORKER CONFIG
master = true
processes = 4 #reduce this to 1 if syscall issue persists
enable-threads = true
#vacuum = true
threads = 8
minspare = 4

When we do not use lazy-apps it appears that it causes this issue. Sometimes it wont happen right away, but once you start writing to the database we start getting flooded with these. It breaks our flask application as well on the front end. A full dump of the error page can be seen here: https://pastebin.com/s5CgBrGs and https://pastebin.com/ySeVivHe
When we turn lazy apps on the ssl syscall issue disappears. Any assistance figuring this out would be greatly appreciated.
Versions.

OS: RHEL 8.3
Python: 3.6
SQLAlchemy: 1.3.23
Flask-SQLAlchemy version: 2.4.4
Database: Postgres 10.13
Flask: 1.1.2
DBAPI:

Sorry, I can't help you debug this, it's way too broad and there's no indication it's related to Flask.
Please use Stack Overflow for questions about your own code. This tracker is for issues related to the project itself. Be sure to include a minimal, complete, and verifiable example.


avoid toolbar side effects

closes #262
prerequisites:

[x] branch is up-to-date with the branch to be merged with, i.e. develop
[x] build is successful
[x] code is cleaned up and formatted
[ ] tested with Firefox 52, Firefox 57+, Chrome 64+, MS Edge 16+

Summary
avoid side effects in the module for caching toolbar actions from columns. When developing toolbar addons one could experience caching issues

@sgratzl Is there a way to test it or what I should have a look it?

to see an effect you would need a project where you create a new column type with a custom toolbar annotation. A running example where the LineUp.js library is loaded and using the new column type. Then when adding a new option to the toolbar and just hot reloading the custom class without loading lineupjs you should see that still the old toolbar is used, since the toolbar options were cached in the library instance of LineUp.js instead of the current LineUp instance.


Updated api documentation to reflect implementation.

/contact/get accepts parameter 'address' not 'id'.
/contact/create parameter 'from' is now documented.

To accept your pull request (even though it's just a couple of changes,) we have to ask you to complete the CLA and email it to<EMAIL_ADDRESS>


Can i choose weapon when in-game ?

HI,  can i choose weapon after i getting it in game or do i have to choose weapon first then start game ?

you choose a weapon first. it would help even if you ends up having a different weapon because the idea to reduce the recoil is the same


Settings keys are not case consistent

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
Some of the settings are using snake case (like keyless_prefix) while others are camel case (like pubKeys).
Everything should stay consistent.
I propose to use camel case because this is what the kubernetes world uses.
Expected Behavior
All the settings keys use the same naming convention
Steps To Reproduce
No response
Environment
No response
Anything else?
No response

It would also be nice to converge to the verificationConfig of PolicyServer (or at least, don't diverge).

which means camelCase. That's fine with me


THRIFT-5199: Fix infinite loop writing to closed TSocket in PHP

Client: php

When you call fwrite in PHP with a handle that has been closed by peer the function will return 0 bytes written (rather than false to indicate an error).
This causes an infinite loop today because the code assumes 0 only means a temporary failure to write, rather than a permanent one.
This change adds a check to for a closed socket when 0 bytes are written

[ X] Did you create an Apache Jira ticket?  (not required for trivial changes)
[ X] If a ticket exists: Does your pull request title follow the pattern "THRIFT-NNNN: describe my issue"?
[ X] Did you squash your changes to a single commit?  (not required, but preferred)
[ X] Did you do your best to avoid breaking changes?  If one was needed, did you label the Jira ticket with "Breaking-Change"?
[ ] If your change does not involve any code, include [skip ci] anywhere in the commit message to free up build resources.

Plus, I found this statement in the comments at

After having problems with fwrite() returning 0 in cases where one would fully expect a return value of false, I took a look at the source code for php's fwrite() itself. The function will only return false if you pass in invalid arguments. Any other error, just as a broken pipe or closed connection, will result in a return value of less than strlen($string), in most cases 0.

If that is true then both the docs and all code relying on it are broken.
So why don't we just check for $written being less/equal 0 instead of === -1) ? Wouldn't that suffice?

I saw that comment on php.net and verified it was true (both by looking at the c implementation of php's fwrite, and by writing a simple test.
I believe that checking for written <= 0 would work and would be more robust.  If there is a closed duplex socket where data was sent from the peer before closing then fwrite will return 0 (the socket is closed) but feof will return false because there is still data to read.
In the patch we are using locally I didn't go that route for two reasons:

I'm don't know enough about the internals of sockets to know if there are some cases where a write of 0 is temporary and a subsequent write would be successful.
The implementation of TSocket::read uses the 0 length + feof check so I erred on the side of consistency.

`if ($readable > 0) {
$data = fread($this->handle_, $len);
if ($data === false) {
throw new TTransportException('TSocket: Could not read ' . $len . ' bytes from ' .
$this->host_ . ':' . $this->port_);
} elseif ($data == '' && feof($this->handle_)) {
throw new TTransportException('TSocket read 0 bytes');
}
        return $data;
    }`

I'm pretty sure that throwing an exception on 0 bytes in both cases without the feof check would be correct.  We just decided to be really conservative with our initial change as we make tens of thousands of thrift calls per second.
I can attest that adding the feof check has fixed our issue with stuck php processes.


build fails on Ubuntu

npm ERR! Linux 4.4.0-66-generic
npm ERR! argv "/root/.nvm/versions/node/v5.11.1/bin/node" "/root/.nvm/versions/node/v5.11.1/bin/npm" "run" "build"
npm ERR! node v5.11.1
npm ERR! npm  v3.8.6
npm ERR! code ELIFECYCLE
npm ERR<EMAIL_ADDRESS>build: babel-node tools/webpack.run build
npm ERR! Exit status 137
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>build script 'babel-node tools/webpack.run build'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the apollo-fullstack-starter-kit package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     babel-node tools/webpack.run build
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs apollo-fullstack-starter-kit
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls apollo-fullstack-starter-kit
npm ERR! There is likely additional logging output above.

@hhashmi Seems you are using very outdated node.js version. Because this starter kit uses latest versions of all its dependencies, you should you latest node.js too.

@hhashmi I would recommend installing and updating node.js on your system via NVM:
https://github.com/creationix/nvm

@hhashmi I have tried to reproduce this bug on Ubuntu 16.04 with stock nodejs v4.2.6 and everything works for me. I don't see Exit status 137 error.

Just found out that I'm on a VM and it was due to low amount of memory.
Increased memory of VM and it is working fine.
Thanx a lot for your inputs.


Fix import targets and simplify succinct roles

This commit fixes the bug on rstuf admin import-targets, which wasn't working with RSTUF Worker/API beta version.
The problem is due to the changes in database models for RSTUF Workers.
It also simplifies the way to retrieve the succinct roles.
Closes #297

Did you test this fix with an RSTUF API and RSTUF Worker deployment?

Yes, I tested in a running evironment

I think the failing of the CI is the same as I saw in my pr #284.
I fixed it in the commit https://github.com/repository-service-tuf/repository-service-tuf-cli/pull/284/commits/c594160f0e623e86a4cae94f6c69b83677d8a3e2, but I have some questions regarding this fix as I shared in my following comment: https://github.com/repository-service-tuf/repository-service-tuf-cli/pull/284#issuecomment-1620240870


openai structured output

use openai.beta.chat.completions.parse with gpt-4o-2024-08-06 and a custom structured output to guarantee valid output

see https://openai.com/index/introducing-structured-outputs-in-the-api/

Seems to somehow be less reliable than the previous version... weird
maybe its because gpt-4o-2024-08-06 is less proficient in hebrew?


fix: text search returns everything on empty string

:tada: This PR is included in version 1.91.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Expand Inquiry Resolution field

Expand Inquiry Resolution field to include more text. Also need to make sure that it can accept formatted text

expanded. just need formatting now.


[BEAM-7426] Add underscore support on Field lexer

Underscore is a common used word delimiter in field names, the current
FieldSpecifierNotationLexer only support alpha-numeric values for field
name character.
The upcoming Protobuf schema support will emit underscores in the field
names, so field names should support underscore.

@reuvenlax this PR would make it possible to use proto field names as they contain underscores in general. This would make the following code work:
            .apply(Filter.<FooBar>create().whereFieldName("test_name", u -> u.equals("abc")))
            .apply(Select.fieldNames("test_name"))
            .apply(ParDo.of(new DoFn<Row, String>() {

It seems that Filter works, but Select doesn't without the fix. Even so, I think the lexer is too restrictive and needs to be revised later on.