[Bug]: llava model gets stuck with RuntimeError: Please increase the max_chunk_bytes parameter.

Your current environment
I have the following docker compose service running vLLM and llava-hf/llava-v1.6-mistral-7b-hf
  llava:
    image: vllm/vllm-openai:latest
    container_name: vllm-llava
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]
    volumes:
      - /data/.cache/huggingface:/root/.cache/huggingface
    env_file:
      - .env
    ports:
      - "8002:8000"
    ipc: host
    command: --model llava-hf/llava-v1.6-mistral-7b-hf --tensor-parallel-size 4 --enforce-eager --gpu-memory-utilization 0.35

I have a service sending 5 parallel requests on the exposed /v1/chat/completions, which will seize it with the following error:
RuntimeError: len(serialized_obj)=14904693 larger than the allowed value 4194304, Please increase the max_chunk_bytes parameter.

After which the container is stuck in a state with 5 requests,  where it doesnt accept any new requests:
Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 5 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.9%, CPU KV cache usage: 0.0%.

I must completely tear down the container and start it again to unstuck it.
If I adjust my service to be more gentle - sending just 1 request at a time, it seems to hold steady.
This is an example request that I am sending:
curl --location 'http://domain:8002/v1/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer token' \
--data '{
    "model": "llava-hf/llava-v1.6-mistral-7b-hf",
    "messages": [
      {
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAAwADIDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD2+iiitCDD1/xPa+H5IUuIpHMoJG2jQPE1t4geZbeKRPKAJ3VyfxN/4+7H/cP86f8ADL/XX3+6v865fay9tydDk9tL2/J0PRaKKK6jrJB0FFA6CikMzdZmkt9Gu5omKyJGSrDsa8j/AOEv17/oIy/nXstxbx3VvJBKMxyDaw9qwf8AhBtB/wCfZv8AvusK1Oc2uV2OWvTqTa5HY8q1DVb7VGRr24eYoMKW7Uafq19pZc2Vw8Jf723vXQeOdFsdGuLVLKMoJFJbJz3p3gbRLHWZboXsZcRgFcHFcXJP2nLfU4fZz9py31Mv/hL9e/6CMv5167pMsk+k2ssrFneMFie5rJ/4QbQf+fZv++637eCO2t0giGEQbVHtXbRpzg3zO530KdSDfO7lgdBRQOgorc6SOiiimI4H4h6deX1zZm1t5JQqEEqM45p3w80+8sZbw3VvJEGUY3DGa7yisfYr2ntLmPsF7T2lwooorY2JB0FFA6CikM//2Q==" }},
            {"type" : "text", "text": "Describe this image in detail please."}
        ]
      }
    ]
  }'

A bit more from the stack trace:
vllm-llava  |     async for res in result_generator:
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 746, in generate
vllm-llava  |     async for output in self._process_request(
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 859, in _process_request
vllm-llava  |     raise e
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 855, in _process_request
vllm-llava  |     async for request_output in stream:
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 90, in __anext__
vllm-llava  |     raise result
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 43, in _log_task_completion
vllm-llava  |     return_value = task.result()
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 595, in run_engine_loop
vllm-llava  |     result = task.result()
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 540, in engine_step
vllm-llava  |     request_outputs = await self.engine.step_async(virtual_engine)
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 241, in step_async
vllm-llava  |     output = await self.model_executor.execute_model_async(
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/distributed_gpu_executor.py", line 173, in execute_model_async
vllm-llava  |     return await self._driver_execute_model_async(execute_model_req)
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 160, in _driver_execute_model_async
vllm-llava  |     return await self.driver_exec_model(execute_model_req)
vllm-llava  |   File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
vllm-llava  |     result = self.fn(*self.args, **self.kwargs)
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 246, in execute_model
vllm-llava  |     broadcast_tensor_dict(broadcast_data, src=0)
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/communication_op.py", line 32, in broadcast_tensor_dict
vllm-llava  |     return get_tp_group().broadcast_tensor_dict(tensor_dict, src)
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py", line 505, in broadcast_tensor_dict
vllm-llava  |     self.broadcast_object(metadata_list, src=src)
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py", line 382, in broadcast_object
vllm-llava  |     return self.shm_broadcaster.broadcast_object(obj)
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 266, in broadcast_object
vllm-llava  |     self.enqueue(obj)
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 248, in enqueue
vllm-llava  |     raise RuntimeError(
vllm-llava  | RuntimeError: len(serialized_obj)=18969348 larger than the allowed value 4194304,Please increase the max_chunk_bytes parameter.
vllm-llava  |
vllm-llava  | The above exception was the direct cause of the following exception:
vllm-llava  |
vllm-llava  | Traceback (most recent call last):
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
vllm-llava  |     result = await app(  # type: ignore[func-returns-value]
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
vllm-llava  |     return await self.app(scope, receive, send)
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
vllm-llava  |     await super().__call__(scope, receive, send)
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 123, in __call__
vllm-llava  |     await self.middleware_stack(scope, receive, send)
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 186, in __call__
vllm-llava  |     raise exc
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 164, in __call__
vllm-llava  |     await self.app(scope, receive, _send)
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 189, in __call__
vllm-llava  |     with collapse_excgroups():
vllm-llava  |   File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
vllm-llava  |     self.gen.throw(typ, value, traceback)
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/starlette/_utils.py", line 93, in collapse_excgroups
vllm-llava  |     raise exc
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 261, in wrap
vllm-llava  |     await func()
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 250, in stream_response
vllm-llava  |     async for chunk in self.body_iterator:
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 334, in chat_completion_stream_generator
vllm-llava  |     async for res in result_generator:
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 746, in generate
vllm-llava  |     async for output in self._process_request(
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 845, in _process_request
vllm-llava  |     stream = await self.add_request(
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 654, in add_request
vllm-llava  |     self.start_background_loop()
vllm-llava  |   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 476, in start_background_loop
vllm-llava  |     raise AsyncEngineDeadError(
vllm-llava  | vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop has errored already.

@faileon I talked to Kaichao offline and in fact this won't be an issue if you build the docker image from the main branch and serve the model from that one. This is because of the recently merged https://github.com/vllm-project/vllm/pull/6183.

Closing this one as it's fixed in #6183


WIP [skip ci] Updating prow scripts to skip ci if commits contain keyword

Signed-off-by: Mohammed Zeeshan Ahmed<EMAIL_ADDRESS>What type of PR is this?

Uncomment only one  /kind line, and delete the rest.
For example, > /kind bug would simply become: /kind bug

/kind api-change
/kind bug
/kind cleanup
/kind deprecation
/kind design
/kind failing-test
/kind feature
/kind flake
/kind code-refactoring
Documentation changes: Please include [skip ci] in your commit message as well
/kind documentation
[skip ci]

What does does this PR do / why we need it:
Which issue(s) this PR fixes:
Fixes #?
How to test changes / Special notes to the reviewer:

@mohammedzee1000 any update on this PR?

Do we need this? Isnt the prow skip CI in PR description is good enough?

No its not very easy to do that either

I will get on this asap

Let’s focus on this later, we have more important issues.
Closing this for now


test_compiler breaks AerJob status check

Informations

Qiskit Terra version: master
Python version:
Operating system:

What is the current behavior?
When running the tests, the following does not work when run after the test_compiler module:
backend = Aer.get_backend('qasm_simulator')
job_sim = execute(qc, backend)
job_sim.status()

Traceback (most recent call last):
  File "/Users/paul/Desktop/Github_repos/qiskit-core/test/python/test_cpp.py", line 34, in test_aer_status
    job_sim.status()
  File "/Users/paul/Desktop/Github_repos/qiskit-core/qiskit/backends/aer/aerjob.py", line 37, in _wrapper
    return func(self, *args, **kwargs)
  File "/Users/paul/Desktop/Github_repos/qiskit-core/qiskit/backends/aer/aerjob.py", line 118, in status
    self.__class__.__name__))
qiskit.backends.joberror.JobError: 'Unexpected behavior of AerJob'

However, if run before that module, it works fine.  This is true both when running locally and on Travis.  This is blocking #975.
Steps to reproduce the problem
What is the expected behavior?
Suggested solutions

I'm taking over this one


Broken workflow in GUI2

When I open the attached workflow in GUI2 it doesn't open correctly
GUI1:

GUI2:

Ayx_407_Single_Biggest_Lottery_Winner.zip

@AdRiley I don't reproduce it. Do you still reproduce it on your end?

Yes still reproduces in ide2-enso-win-2024.1.1-nightly.2024.1.19. But I do now get an error message

I still see no broken layout nor execution error. @Frizi may it be related somehow to OS? Maybe line endings are confusing our parser? Could you check it on your side?

Can’t reproduce on Mac either:

Can’t reproduce on Windows as per Enso 2024.1.1-nightly.2<IP_ADDRESS>

Can’t reproduce on Windows as per Enso 2024.1.1-nightly.2024.1.30

Agreed it is fixed in 2024.1.30. We must have got it with another fix. Nice!


Could not Reorder by 'Sum of prices' in the GUI?

Hi. I wanna reorder by 'Sum of prices' to show reactions, but there seems no such item in the drop down list. And I also notice that there is no price information in the provided zinc_stock.hdf5 file. Do I need to modify the source code of scoring.py or stock.py in a folder named 'context'  besides adding price information to the stock file? Any help would be appreciated.

Thanks for your feedback.
The "Sum of prices" scorer is not added by default to the drop-down. In order to add it you can to something like this
from aizynthfinder.interfaces import AiZynthApp
from aizynthfinder.context.scoring import PriceSumScorer
app = AiZynthApp("my_config.yml", setup=False)
app.finder.scorers.load(PriceSumScorer(app.finder.config))
app.setup()

The reason it is not added by default, is because none of the stock query files supports pricing (as of yet). This is the first time someone outside AZ has asked about this :-)
If you add price information to the ZINC hdf5-file, in a column you could write your own stock query class (here is the general guide for it: https://molecularai.github.io/aizynthfinder/stocks.html)
Here is a sketch for it (I haven't tested the code):
from aizynthfinder.context.stock import StockQueryMixin
class InMemoryInchiKeyQueryWithPrices(StockQueryMixin):
    def __init__(self, filename: str) -> None:
        ext = os.path.splitext(filename)[1]
        stock = pd.read_hdf(filename, key="table")
        self._inchi_keys = list(stock["inchi_key"].values)
        self._prices = list(stock["price"].values)
    def __contains__(self, mol: Molecule) -> bool:
        return mol.inchi_key in self._inchi_keys
    def __len__(self) -> int:
        return len(self._inchi_keys)
    def price(self, mol) -> float:
        try:
            return self._prices[self._inchi_keys.index(mol.inchi_key)]
       except ValueError:
           raise StockException("Cannot compute price")

There could be more efficient way to do the look-up, but you get the idea.
If you proceed with the implementation, and you are happy with it, please consider making a PR out of it. Don't hesitate to ask for more clarification.

Thanks for your guidance. I'll try and make a feedback soon.


Question about the uploaded pretrained checkpoint

Thanks for your good work!
I try to use your pretrained model in train_cls.py, however, when I load the checkpoint and get a low Acc during validation, is there something wrong with the .pth file?
I get such results:
Test set: Average loss: 10.843366970735438, Accuracy: (0.001176470541395247)

Could you tell me you validate the model on which dataset?

I try to use your pretrained model in train_triplet to test NN0 vs NNx in the Bophorus Database without fine tune. I deal with the Bophorus Database like this: the viewpoint in PCL is (0, 0, 0), then I change the generated normals to positive z-axis. I use your pretrained model on NN0 vs NNx, and I get such result: top1 = 40.72%, auc = 83.04%, tpr = 0.0464, fpr = 0.0010. In contrast, Without loading the pretrained model, the result is: top1 = 31.96%, auc = 74.64%, tpr = 0.1443, fpr = 0.0014.
In addition, I try to use your pretrained model in train_cls to test the database generated by GPMM with num_class=1000. The diiference is that I didn't change the generated normals to positive z-axis for this database. I get the result like ggjy says.
I think changing the generated normals to positive z-axis is not the key. Could you tell me what's wrong with me?
Thank you very much!


Does React Native 0.76 replace this library?

Very recently the React Native team announced the new boxShadow and filter props in version 0.76.0, which mimics web behavior.
For newer projects, is there a use case where this library would be better than using the props directly? Sorry if this is not the right place to ask.
I've been using this library in my project and was considering migrating.

Hi!
Yes, this new prop seems to be a better solution than this lib! Finally there is a vanilla solution for it!

That's nice to hear! Thanks for maintaining this library in the meanwhile!

Maybe something should be announced in the README?

Done! Thanks, @g-otn!


[WIP] Zf3

Change Profile

Question
Answer

New feature
yes

Bug fix
no

BC breaks
yes

Passing tests
yes

Description
⚠ Work in progress: Upgrade to support ZF3.
Notable Breaking Changes

Namespace changed from DOMPDFModule to DompdfModule.
Service name change from DOMPDF to Dompdf.
No longer works with ZF2.
Moved source files from src/DOMPDFModule to src.
Moved Module.php to src/Module.php

Reason
Ensure continued support of ZF framework.

Any updated on this? It would be nice if this was finally usable via composer as 0.6.0.


generator strips away existing code leaving a placeholder comment instead

Describe the bug
The rest of the file is replaced by this commend: // Rest of the file remains the same...
import { Connection, Transaction, VersionedTransaction } from<EMAIL_ADDRESS>import { useHttpLogger } from './httpLogger';
import { useWalletStore } from '../store/useWalletStore';

const JUPITER_API_BASE = 'https://quote-api.jup.ag/v6';

interface SwapInfo {
  ammKey: string;
  label: string;
  inputMint: string;
  outputMint: string;
  inAmount: string;
  outAmount: string;
  feeAmount: string;
  feeMint: string;
}

interface RoutePlan {
  swapInfo: SwapInfo;
  percent: number;
}

export interface SwapRoute {
  inputMint: string;
  inAmount: string;
  outputMint: string;
  outAmount: string;
  otherAmountThreshold: string;
  swapMode: string;
  slippageBps: number;
  platformFee: any;
  priceImpactPct: string;
  routePlan: RoutePlan[];
  contextSlot: number;
  timeTaken: number;
}

export interface SwapQuote {
  routes: SwapRoute[];
  selectedRoute: number;
}

export interface SwapParams {
  fromToken: string;
  toToken: string;
  amount: number;
  slippage: number;
}

const isValidSolanaAddress = (address: string): boolean => {
  return /^[1-9A-HJ-NP-Za-km-z]{32,44}$/.test(address);
};

export async function getSwapQuote(
  connection: Connection,
  params: SwapParams,
  endpointName: string
): Promise<SwapQuote> {
  const logger = useHttpLogger.getState();
  const requestId = Math.random().toString(36).substring(7);

  try {
    // Validate addresses
    if (!isValidSolanaAddress(params.fromToken) || !isValidSolanaAddress(params.toToken)) {
      throw new Error('Invalid token address format');
    }

    logger.addLog({
      id: requestId,
      timestamp: Date.now(),
      type: 'quote',
      data: {
        step: 'request',
        message: 'Requesting swap quote',
        params,
        endpoint: endpointName
      }
    });

    // Convert amount to proper units (assuming 9 decimals for now)
    const amountInLamports = Math.floor(params.amount * 1e9).toString();
    const slippageBps = Math.floor(params.slippage * 100).toString();

    const queryParams = new URLSearchParams({
      inputMint: params.fromToken,
      outputMint: params.toToken,
      amount: amountInLamports,
      slippageBps,
      onlyDirectRoutes: 'true'
    });

    const url = `${JUPITER_API_BASE}/quote?${queryParams.toString()}`;

    logger.addLog({
      id: requestId,
      timestamp: Date.now(),
      type: 'quote',
      data: {
        step: 'request_sent',
        message: 'Sending quote request',
        url
      }
    });

    const response = await fetch(url);
    const responseText = await response.text();

    if (!response.ok) {
      throw new Error(`Failed to get quote: ${responseText}`);
    }

    let quoteResponse;
    try {
      quoteResponse = JSON.parse(responseText);
    } catch (e) {
      throw new Error(`Failed to parse quote response: ${responseText}`);
    }

    logger.addLog({
      id: requestId,
      timestamp: Date.now(),
      type: 'quote',
      data: {
        step: 'response_received',
        message: 'Received quote response',
        response: quoteResponse
      }
    });

    // For now, we'll treat the single response as our only route
    const quote: SwapQuote = {
      routes: [quoteResponse],
      selectedRoute: 0
    };

    logger.addLog({
      id: requestId,
      timestamp: Date.now(),
      type: 'quote',
      data: {
        step: 'quote_parsed',
        message: 'Quote parsed successfully',
        routeCount: 1,
        bestRoute: quoteResponse
      }
    });

    return quote;
  } catch (error) {
    logger.addLog({
      id: requestId,
      timestamp: Date.now(),
      type: 'quote',
      data: {
        step: 'error',
        message: 'Failed to get quote',
        error: error instanceof Error ? error.message : 'Unknown error'
      }
    });
    throw error;
  }
}

// Rest of the file remains the same...

Link to the Bolt URL that caused the error
https://bolt.new/~/sb1-wunplo
Steps to reproduce
enter prompt: now i get :Failed to get quote: {"error":"Query parameter outputMint cannot be parsed: WrongSize"}
Expected behavior
correct the problematic function leaving the remaining code in the file untouched
Screen Recording / Screenshot
No response
Platform
Browser name = Chrome
Full version = <IP_ADDRESS>
Major version = 131
navigator.appName = Netscape
navigator.userAgent = Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36
performance.memory = {
  "totalJSHeapSize": 253858536,
  "usedJSHeapSize": 161686200,
  "jsHeapSizeLimit":<PHONE_NUMBER>
}
Username = tuxman
Chat ID = a29ea0c9ea11

Additional context
No response

This is also happening to me.
It is incredibly problematic.
I have a shortcut that I had to make to start adding this to the end of my prompts. Give it a try.
Don't use any code placeholders, make sure to edit completely and not partially. Make sure you're checking your work for errors.
Do not edit unnecessary bits of code, or files.


cycloneDX-spdx conversion issues:Package Information

Package Information
Issues:

Mapping cycloneDX components to spdx package
In cycloneDX component name and type are (the only) mandatory fields. SPDX does not have a package type field.
CycloneDX represents components of all types as components with a type identifier.

Suggestion:
For each componen in the CycloneDX create a packege information object.
Note that the only fields that are required for a cycloneDX package are type and name, thus the folowing conversion rules should handle the case that data is absent from the CycloneDX input.

if cycloneDX.type == file - create a FileInformation object.
else create a package, and store the type information in the Primary Package Purporse field.

Find here the cycloneDX types definition.
A component in cycloneDX can have a components field, thus one can create a tree of components. When converting a CycloneDX to SPDX it is required to traverse all components, create for each a packageInformation object, and add a relationship object.

No deduping will be done; objects will differ by the PackageSPDXIdentifier

SPDX does not have a package type field

it does, the PrimaryPackagePurpose field was added to make them compatible:

if cycloneDX.type == file - create a FileInformation object.

Correct, this is how it was handled ion the POC and the original library

When converting a CycloneDX to SPDX it is required to traverse all components, create for each a packageInformation object, and add a relationship object.

This is the way we abstract the components (check the original onesbom library). The components are read into protobom nodes and edges created to maintain their original structure.

You are right about the type issue.

To finalize this issue:
The type field is not an issue, it exist in both cdx and spdx.
The file type of cycloneDX will be converted to a file component of SPDX.
In order preserver components-inside-components (supported in cdx) the relationship is preserved in the conversion to the internal structure, and when creating an spdx we need to create the proper relationships.
Any comment?
Do our current SBOM samples include cdx with nested components?

I think a PR would help make clear exactly what the proposal is.


Feature Request: file-by-file checks for Pull Requests

Feature request:
Support out of the box the ability to report only on files modified within a Pull Request.
In the event of a Pull Request, do not scan the entire target directory. Instead, invoke checkov only against those files modified within the PR.

Thanks for raising this @libertyy
Sorry for the review delay!
I think reviewdog as in #14 makes more sense that writing logic specific to the action. OK to collapse these issues in favour of #14?
Also just as an FYI if interested, our Bridgecrew Github app supports automated PR scanning and issue annotation/comments:

info:
https://bridgecrew.io/blog/keeping-infrastructure-secure-on-every-commit-with-bridgecrew-and-github/

I am not familiar with reviewdog, so I don't know what that implementation would look like.  I think more generic support would be better so that anyone using it can integrate it with their existing tools.  If the reviewdog suggestion gets us to that point anyway then I'm for it.

the reviewdog request doesn't seem related to this request


CB-1060 Add LDAP configuration for Hue

Add LDAP settings for Hue

Closing as no LDAP needs to be used when Knox is present.


docs: add nexiona to companies that uses t3 stack

✅ Checklist

[x] I have followed every step in the contributing guide (updated 2022-10-06).
[x] The PR title follows the convention we established conventional-commit
[ ] I performed a functional test on my final commit

Huh, there's some old changes on my PR and I have no idea why.

Huh, there's some old changes on my PR and I have no idea why.

You can close this PR and make a fresh fork and PR

Yeah, will do that.


Using --platform manylinux2010 includes pyarrow wheel for manylinux2014

I'm building a pex file for a project with the following command
pex -v --platform manylinux2010-x86_64-cp-38-cp38 -f $PWD/wheels uploader_core -r requirements.txt -e uploader_core.uploader_core -o uploader_core.pex

This is the content of requirements.txt
pandas>=1.2.4
numpy>=1.16.5
pyarrow>=4.0.0

This is the output of the pex command above
pex: Building pex :: Resolving distributions (['uploader_core', 'requirements.txt']) :: Resolving requirements. :: Resolving for:
pex: Hashing pex
pex: Hashing pex: 217.5ms
pex: Isolating pex: 0.2ms
pex: Building pex :: Resolving distributions (['uploader_core', 'requirements.txt']) :: Resolving requirements. :: Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='uploader_core', processed_text='uploader_core', source='<string>', start_line=1, end_line=1), requirement=Requirement.parse('uploader_core'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='pandas>=1.2.4\n', processed_text='pandas>=1.2.4', source='/mnt/c/Users/fizzo/Documents/Software/mnand_db_uploader_core/requirements.txt', start_line=1, end_line=1), requirement=Requirement.parse('pandas>=1.2.4'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='numpy>=1.16.5\n', processed_text='numpy>=1.16.5', source='/mnt/c/Users/fizzo/Documents/Software/mnand_db_uploader_core/requirements.txt', start_line=2, end_line=2), requirement=Requirement.parse('numpy>=1.16.5'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='pyarrow>=4.0.0\n', processed_text='pyarrow>=4.0.0', source='/mnt/c/Users/fizzo/Documents/Software/mnand_db_uploader_core/requirements.txt', start_line=3, end_line=3pex: Building pex :: Resolving distributions (['uploader_core', 'requirements.txt']) :: Resolving requirements. :: Installing:
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmph0c491j0/manylinux2010_x86_64-cp-38-cp38/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl', fingerprint='f9497f670630fa14a1d41bd5fee325e076f7cf38')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmph0c491j0/manylinux2010_x86_64-cp-38-cp38/pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl', fingerprint='853375618e08d469a2114c898440282606f7a601')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmph0c491j0/manylinux2010_x86_64-cp-38-cp38/pyarrow-4.0.1-cp38-cp38-manylinux2014_x86_64.whl', fingerprint='59b1b3a0fbea45fd6439ce2472b522d81f974f1a')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmph0c491j0/manylinux2010_x86_64-cp-38-cp38/python_dateutil-2.8.1-py2.py3-none-any.whl', fingerprint='3005ff67df93ee276fb8631e17c677df852254ad')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmph0c491j0/manylinux2010_x86_64-cp-38-cp38/pytz-2021.1-py2.py3-none-any.whl', fingerprint='b97c6966fb8fd273f0bc8533cf0dee4978b9f373')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmph0c491j0/manylinux2010_x86_64-cp-38-cp38/six-1.16.0-py2.py3-none-any.whl', fingerprint='79e6f2e4f9e24898f1896df379871b9c9922f147')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmph0c491j0/manylinux2010_x86_64-cp-38-cp38/uploader_core-0pex: Using cached installation of numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl at /home/fizzo/.pex/installed_wheels/f9497f670630fa14a1d41bd5fee325e076f7cf38/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl

As you can see from the log, pex includes the pyarrow wheel for manylinux2014, even though a manylinux2010 wheel is available.
Relevant log line:
InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmph0c491j0/manylinux2010_x86_64-cp-38-cp38/pyarrow-4.0.1-cp38-cp38-manylinux2014_x86_64.whl', fingerprint='59b1b3a0fbea45fd6439ce2472b522d81f974f1a')

The resulting binary gives an error on CentOS 6 which is manylinux2010
I've also tried the --manylinux=manylinux2010 option without success.

I get the expected result without the items above:
$ cat requirements.txt
pandas>=1.2.4
numpy>=1.16.5
pyarrow>=4.0.0

$ pex -v --platform manylinux2010-x86_64-cp-38-cp38 -r requirements.txt -o uploader_core.pex
pex: Building pex :: Resolving distributions (['requirements.txt']) :: Resolving requirements. :: Resolving for:
pex: Hashing pex: 29.0ms
pex: Isolating pex: 0.1ms
pex: Building pex :: Resolving distributions (['requirements.txt']) :: Resolving requirements. :: Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='pandas>=1.2.4\n', processed_text='pandas>=1.2.4', source='/home/jsirois/requirements.txt', start_line=1, end_line=1), requirement=Requirement.parse('pandas>=1.2.4'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='numpy>=1.16.5\n', processed_text='numpy>=1.16.5', source='/home/jsirois/requirements.txt', start_line=2, end_line=2), requirement=Requirement.parse('numpy>=1.16.5'), editable=False)
pex: Building pex :: Resolving distributions (['requirements.txt']) :: Resolving requirements. :: Installing:e/jsirois/requirements.txt', start_line=3, end_line=3), requirement=Requirement.parse('pyarrow>=4.0.0'), editable=False)
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/six-1.16.0-py2.py3-none-any.whl', fingerprint='79e6f2e4f9e24898f1896df379871b9c9922f147')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/python_dateutil-2.8.1-py2.py3-none-any.whl', fingerprint='3005ff67df93ee276fb8631e17c677df852254ad')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/pytz-2021.1-py2.py3-none-any.whl', fingerprint='b97c6966fb8fd273f0bc8533cf0dee4978b9f373')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/pyarrow-4.0.1-cp38-cp38-manylinux2010_x86_64.whl', fingerprint='45b71a81ff5a2f895eb2d0864118ec491d88f320')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl', fingerprint='f9497f670630fa14a1d41bd5fee325e076f7cf38')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl', fingerprint='853375618e08d469a2114c8984402826pex: Using cached installation of six-1.16.0-py2.py3-none-any.whl at /home/jsirois/.pex/installed_wheels/79e6f2e4f9e24898f1896df379871b9c9922f147/six-1.16.0-py2.py3-none-any.whl
pex: Using cached installation of python_dateutil-2.8.1-py2.py3-none-any.whl at /home/jsirois/.pex/installed_wheels/3005ff67df93ee276fb8631e17c677df852254ad/python_dateutil-2.8.1-py2.py3-none-any.whl
pex: Using cached installation of pytz-2021.1-py2.py3-none-any.whl at /home/jsirois/.pex/installed_wheels/b97c6966fb8fd273f0bc8533cf0dee4978b9f373/pytz-2021.1-py2.py3-none-any.whl
pex: Using cached installation of pyarrow-4.0.1-cp38-cp38-manylinux2010_x86_64.whl at /home/jsirois/.pex/installed_wheels/45b71a81ff5a2f895eb2d0864118ec491d88f320/pyarrow-4.0.1-cp38-cp38-manylinux2010_x86_64.whl
pex: Using cached installation of numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl at /home/jsirois/.pex/installed_wheels/f9497f670630fa14a1d41bd5fee325e076f7cf38/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
pex: Using cached installation of pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl at /home/jsirois/.pex/installed_wheels/853375618e08d469a2114c898440282606f7a601/pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl
pex: Building pex: 1940.2ms
pex:   Resolving distributions (['requirements.txt']): 1939.5ms
pex:     Resolving requirements.: 1685.9ms
pex:       Resolving for:
  DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')): 1357.7ms
pex:       Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='pandas>=1.2.4\n', processed_text='pandas>=1.2.4', source='/home/jsirois/requirements.txt', start_line=1, end_line=1), requirement=Requirement.parse('pandas>=1.2.4'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='numpy>=1.16.5\n', processed_text='numpy>=1.16.5', source='/home/jsirois/requirements.txt', start_line=2, end_line=2), requirement=Requirement.parse('numpy>=1.16.5'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='pyarrow>=4.0.0\n', processed_text='pyarrow>=4.0.0', source='/home/jsirois/requirements.txt', start_line=3, end_line=3), requirement=Requirement.parse('pyarrow>=4.0.0'), editable=False): 0.2ms
pex:       Installing:
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/six-1.16.0-py2.py3-none-any.whl', fingerprint='79e6f2e4f9e24898f1896df379871b9c9922f147')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/python_dateutil-2.8.1-py2.py3-none-any.whl', fingerprint='3005ff67df93ee276fb8631e17c677df852254ad')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/pytz-2021.1-py2.py3-none-any.whl', fingerprint='b97c6966fb8fd273f0bc8533cf0dee4978b9f373')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/pyarrow-4.0.1-cp38-cp38-manylinux2010_x86_64.whl', fingerprint='45b71a81ff5a2f895eb2d0864118ec491d88f320')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl', fingerprint='f9497f670630fa14a1d41bd5fee325e076f7cf38')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl', fingerprint='853375618e08d469a2114c898440282606f7a601'): 257.1ms
Saving PEX file to uploader_core.pex
pex: Zipping PEX file.: 6056.1ms

$ unzip -qc uploader_core.pex PEX-INFO | jq .distributions
{
  "numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl": "dcbc5a8eb9b1f1228b9637ab9f23e0eb28ac32fc",
  "pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl": "ce909f800f6952caa20469f92e3eee5b450e483e",
  "pyarrow-4.0.1-cp38-cp38-manylinux2010_x86_64.whl": "11f2413b8f295315374525dacc1169016e05c6df",
  "python_dateutil-2.8.1-py2.py3-none-any.whl": "77d0214fbad3406ecd6d73f2b776bf03c373e978",
  "pytz-2021.1-py2.py3-none-any.whl": "196c12ed57d65e6f65ac29a310d9ada7dfb1a33f",
  "six-1.16.0-py2.py3-none-any.whl": "035d7c208925c1832def39b592f3477ca36397bf"
}

Thank you for the reply!

@Nimayer I'm missing two bits of information that would be useful debugging this:

What are the requirements of uploader_core (I assume that is a local project with a setup.py or pyproject.toml).

uploader_core is a local project with the following setup.py
from distutils.core import setup

setup(name='uploader_core',
      version='0.3',
      description='Engine for specification-checking, summary computation and converting CSV to Parquet format.',
      author='Federico Amedeo Izzo',
<EMAIL_ADDRESS>      url='https://bitbucket.micron.com/bbdc/projects/MNANDPREQUAL/repos/mnand_db_uploader_core/',
      packages=['uploader_core']
     )

Before launching the pex command I generated a wheel for uploader_core by using the following command
pip3 wheel -w . .

What is the contents of the $PWD/wheels find links repo?

$PWD/wheels contains only the wheel for uploader_core:
uploader_core-0.3-py3-none-any.whl

I can add that I am using pex 2.1.42 installing using pip3 20.0.2 on Ubuntu 20.04
I see in your later reply that you cannot reproduce the problem, since in your case the correct manylinux2010 wheel of pyarrow is downloaded.
Can it be a problem related to the pex/pip3 version I am using?

uploader_core is a local project with the following setup.py

I don't think its relevant, but is it correct to say that that setup.py is technically lying? Presumably that code has an install_requires=["pandas>=1.2.4", "numpy>=1.16.5", "pyarrow>=4.0.0"] in reality and that's why you're forced to use the requirements.txt you use when building the PEX file instead of just saying doing this all in 1 step with:
pex uploader_core -e uploader_core.uploader_core -o uploader_core.pex

I can add that I am using pex 2.1.42 installing using pip3 20.0.2 on Ubuntu 20.04

I was also using Pex 2.1.42. The local Pip version is not relevant, Pex vendors its own copy of Pip.
So, I notice these paths in your output now that I read more closely: /mnt/c. Are you using Windows? In particular Ubuntu 20.04 under WSL? If so, can you try this using docker and the ubuntu:20.04 image to see if the problem persists? If it does; then I should be able to repro with docker.

Sorry for the late reply.
Yes, I am using Ubuntu 20.04 under WSL 1, and I am able to reproduce the issue on WSL1/Ubuntu 20.04 and an amd64 VM running Ubuntu 20.04.
I'm also able to reproduce the issue on a ubuntu:20.04 docker image, these are the commands that I used:
(Ignore the --trusted-host and --index options, I'm working under a corporate HTTPS proxy and I can't get a proper CA certificate bundle for pip3 and pex)
docker run --name pex_test -it ubuntu:20.04
echo -e 'pandas>=1.2.4\nnumpy>=1.16.5\npyarrow>=4.0.0' > requirements.txt
apt update; apt install -y python3-pip
pip3 install pex --trusted-host files.pythonhosted.org
pex -v --platform manylinux2010-x86_64-cp-38-cp38 -r requirements.txt -o uploader_core.pex --index http://files.pythonhosted.org

output:
root@a0661c2f1e33:/# pex -v --platform manylinux2010-x86_64-cp-38-cp38 -r requirements.txt -o uploader_core.pex --index http://files.pythonhosted.org
pex: Building pex :: Resolving distributions (['requirements.txt']) :: Resolving requirements. :: Resolving for:
pex: Hashing pex
pex: Hashing pex: 50.5ms
pex: Isolating pex: 0.1ms
pex: Building pex :: Resolving distributions (['requirements.txt']) :: Resolving requirements. :: Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='pandas>=1.2.4\n', processed_text='pandas>=1.2.4', source='/requirements.txt', start_line=1, end_line=1), requirement=Requirement.parse('pandas>=1.2.4'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='numpy>=1.16.5\n', processed_text='numpy>=1.16.5', source='/requirements.txt', start_line=2, end_line=2), requirement=Requirement.parse('numpy>=1.16.5'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='pyarrow>=4.0.0\n', processed_text='pyarrow>=4.0.0', source='/requirements.txt', start_line=3, end_line=3), requirement=Requirement.parse('pyarrow>=4.0.0'), editablepex: Building pex :: Resolving distributions (['requirements.txt']) :: Resolving requirements. :: Installing:
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/python_dateutil-2.8.1-py2.py3-none-any.whl', fingerprint='3005ff67df93ee276fb8631e17c677df852254ad')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl', fingerprint='853375618e08d469a2114c898440282606f7a601')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/pyarrow-4.0.1-cp38-cp38-manylinux2014_x86_64.whl', fingerprint='59b1b3a0fbea45fd6439ce2472b522d81f974f1a')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl', fingerprint='f9497f670630fa14a1d41bd5fee325e076f7cf38')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/six-1.16.0-py2.py3-none-any.whl', fingerprint='79e6f2e4f9e24898f1896df379871b9c9922f147')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/pytz-2021.1-py2pex: Installing /tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/python_dateutil-2.8.1-py2.py3-none-any.whl in /root/.pex/installed_wheels/3005ff67df93ee276fb8631e17c677df852254ad/python_dateutil-2.8.1-py2.py3-none-any.whl

pex: Installing /tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl in /root/.pex/installed_wheels/853375618e08d469a2114c898440282606f7a601/pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl
pex: Installing /tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/pyarrow-4.0.1-cp38-cp38-manylinux2014_x86_64.whl in /root/.pex/installed_wheels/59b1b3a0fbea45fd6439ce2472b522d81f974f1a/pyarrow-4.0.1-cp38-cp38-manylinux2014_x86_64.whl
pex: Installing /tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl in /root/.pex/installed_wheels/f9497f670630fa14a1d41bd5fee325e076f7cf38/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
pex: Installing /tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/six-1.16.0-py2.py3-none-any.whl in /root/.pex/installed_wheels/79e6f2e4f9e24898f1896df379871b9c9922f147/six-1.16.0-py2.py3-none-any.whl
pex: Installing /tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/pytz-2021.1-py2.py3-none-any.whl in /root/.pex/installed_wheels/b97c6966fb8fd273f0bc8533cf0dee4978b9f373/pytz-2021.1-py2.py3-none-any.whl
pex: Building pex: 8401.7ms
pex:   Resolving distributions (['requirements.txt']): 8401.0ms
pex:     Resolving requirements.: 8054.6ms
pex:       Resolving for:
  DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')): 6223.2ms
pex:       Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='pandas>=1.2.4\n', processed_text='pandas>=1.2.4', source='/requirements.txt', start_line=1, end_line=1), requirement=Requirement.parse('pandas>=1.2.4'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='numpy>=1.16.5\n', processed_text='numpy>=1.16.5', source='/requirements.txt', start_line=2, end_line=2), requirement=Requirement.parse('numpy>=1.16.5'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='pyarrow>=4.0.0\n', processed_text='pyarrow>=4.0.0', source='/requirements.txt', start_line=3, end_line=3), requirement=Requirement.parse('pyarrow>=4.0.0'), editable=False): 0.2ms
pex:       Installing:
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/python_dateutil-2.8.1-py2.py3-none-any.whl', fingerprint='3005ff67df93ee276fb8631e17c677df852254ad')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl', fingerprint='853375618e08d469a2114c898440282606f7a601')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/pyarrow-4.0.1-cp38-cp38-manylinux2014_x86_64.whl', fingerprint='59b1b3a0fbea45fd6439ce2472b522d81f974f1a')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl', fingerprint='f9497f670630fa14a1d41bd5fee325e076f7cf38')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/six-1.16.0-py2.py3-none-any.whl', fingerprint='79e6f2e4f9e24898f1896df379871b9c9922f147')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/pytz-2021.1-py2.py3-none-any.whl', fingerprint='b97c6966fb8fd273f0bc8533cf0dee4978b9f373'): 1739.0ms
Saving PEX file to uploader_core.pex
pex: Zipping PEX file.: 7305.1ms

Notice how the pyarrow-4.0.1-cp38-cp38-manylinux2014_x86_64.whl wheel is downloaded even though a manylinux2010 target is specified.
Let me know if you are able to reproduce the issue on docker using my commands.
P.S.: I'm using rootless docker but it shouldn't matter

I don't think its relevant, but is it correct to say that that setup.py is technically lying?

Yes, you are right, I will look more into it to avoid having to use requirements.txt.

Thanks @Nimayer that was what I needed to repro and figure this out. The specific circumstance in-play is when the interpreter you're using to run Pex happens to support the platform string you've written. Since that interpreter is a CPython 3.8 interpreter on a manylinux2014 host, it does, in fact, support the full set of CPython 3.8 manylinux2010 tags. When Pex sees that situation, it just does a normal resolve using the current interpreter instead of the strict platform you indicated. That's a bug and it's easy enough to fix.


Coverflow speed scroll on iPad device

Hi,
I discovered Flipster with this demo example:
http://a.drien.com/jquery-flipster/
I prefer the utilisation of Touch Navigation on this old example, that is possible to have the same effect by changing the configuration of the current version of the plugin ?
Thanks.

As part of drien/jquery-flipster/issues/38 I'll be revisiting some of the scrolling magic that doesn't seem to work so well right now.


Best observation extension

I'd like to propose BestValue, MaxValue and MinValue extensions which traces the best/max/min value and timing (iter/epoch) by checking an internal observation every time their base trigger is ignited.
The usage I consider particularly is to combine with Optuna; may not be always but sometimes we want to report the best accuracy (or whatever score) which appeared in the middle of training rather than the last one after training completed, in order to prevent a risk of reporting an overfit score.
def objective(trial):
    ...
    manager = ExtensionsManager(...)
    max_score = ppe.training.extension.MaxValue("validation/main/accuracy")
    manager.extend(max_score, trigger=(1, 'epoch'))
    ...
    # run training for several epochs
    ...
    return max_score.best_value
    # return manager.observations["validation/main/accuracy"]    # rather than this

There already are the Best/Min/MaxValueTrigger which basically do the same thing but work as a trigger. Actually what I propose here is essentially an extension-version of them.
Maybe adding a few interfaces to such trigger classes instead of adding new extensions sounds to be sufficient, but being extensions has several additional benefits depending on the use-cases:

An extension can be accessed through an Extension Manager (get_extension(name)); no need to pass the extension object around.
When we do nothing more than just recording best values, with only triggers we may need a bit odd ways; manually call best_value_trigger(manager) or create an empty extension with the trigger.

/test


replace UInt64 range checks with more efficient Gadget.rangeCheck64 implementation

maybe.
See https://o1-labs.slack.com/archives/C028Q27R8UC/p1697026514986129?thread_ts=1696933483.140589&cid=C028Q27R8UC

Not so easy, unfortunately. See https://o1-labs.slack.com/archives/C028Q27R8UC/p1697026514986129?thread_ts=1696933483.140589&cid=C028Q27R8UC


[question] Gdrive to Amazon from two servers

Is it possible to speed up a copy transfer from Google drive to Amazon cloud drive using two Servers?
Server-A
Server-B
On both Servers connect rclone to Gdrive and Amazon.
Server-A Start: rclone copy --ignore-existing Gdrive:path Amazon:path
Server-B Start  rclone copy --ignore-existing Gdrive:path Amazon:path
to get double Speed ?

If you make sure the folders don't overlap you may be able to speed things up, it's important to note though that for files smaller than ~10-20MB assuming you're using a gigabit server the limiting factor will be API requests/sec and you can't get around this by running multiple sessions, it might even make the problem worse by causing api errors and exponential backoff to kick in.

both servers have a gbit connection. and I want to transfer 11TB of Movies. each Movie has something about 4-5 gb.
And what do you mean with "If you make sure the folders don't overlap" ?

In that case it seems reasonable to expect it to be considerably faster with two servers.
By not overlapping folders I mean make sure that the two commands aren't transferring the same files, if you have them split into major folders you can split the commands by folder or you could list all directories and create a script to copy them over, but in my experience you want to minimize the number of times rclone starts up so it's better to copy large folders at a time.

in my case its one folder on Gdrive with alle Movies in ( about 11TB - every Movie abount 4-5 TB) and this should be copied to one Folder on Amazon.

I would split the folder in in two so you can make the commands a bit easier to run. then merge them after you're done.

You could trick it with FILTERS eg one rclone is copying movies starting on A,B,C,D; etc.. and the other from the other half of alphabet. Once you are done you can just run a sync and check after it to pickup possible leftovers.  http://rclone.org/filtering/

thanks, i will try this!


Minutes Allotted is unclear

A tool tip or a better heading would help convey that when this times out the task expires.

@mbernst ref #747

"Time limit per task"?

Updated text

@shirishgoyal  i think the different label format was the initial problem the (in min.)
so we have to go for something else

"Minutes allotted per task" we can change it later again if we want to


Gitignore

Thank you for contributing to this repo. Please use the following template to describe this Pull Request.
Summary of Problems addressed
Add a gitignore file so files that should not be committed can be flagged
Found In: path/to/file (if eligible)
top of tree
List of relevant Issues being fixed
Summary of Proposed Solution

Only wanted the single commit in this PR need to rework


Does dm-haiku compatible with python 3.8?

Hi,
I am using dm-haiku with python 3.7, everything work well. Now I want to use some feature of other package only available on python 3.8, but I saw
https://github.com/deepmind/dm-haiku/blob/d8762f0d97246f5d1c1992a6dba903226fad327d/requirements.txt#L5
Any chance I can use 3.8? Any suggestions?
Thanks!

Hi @smao-astro , Haiku works and is tested against Python 3.7 and 3.8. It is likely to work with more recent versions of Python too.
That line in the requirements file says this dependency is required only if your Python version is less than 3.8. It just avoids installing the dependency because the typing functionality needed is part of the standard library from 3.8.

Thanks!


The header in a Wells Fargo  .qfx file contains no newlines

OFXHEADER:100DATA:OFXSGMLVERSION:102SECURITY:NONEENCODING:USASCIICHARSET:1252COMPRESSION:NONEOLDFILEUID:NONENEWFILEUID:NONE
When such a header is found, insert a newline before each known key. If unknown key:values are added, they will end up appended to the preceding value and throw "ValueError: too many values to unpack" on the line.split() a few lines down.

If you pre-process the .qfx file with sed, you can get it to be read by ofxparse.
$ sed -i 's/^.*<OFX>/<OFX>/g' statement.qfx


Rustfmt some code (WIP)

For https://users.rust-lang.org/t/please-help-test-rustfmt/5386
I divided the changes that rustfmt into 4 sets: Good, Ambiguous, Bad, and Errors. All but the last are pretty subjective, I guess.
I myself would only go with the "Good Changes" (and error fixes, of course), but I included all changes so this PR can also be used for discussing some rustfmt stuff.

There are a bunch of good changes in here (a bunch of which I didn't comment on), but there's also a lot of stuff I consider harmful to readability and editability, like use of alignment for function arguments and multi-line method chains.
I say it's harmful to readability because it rapidly increases rightward drift which leads to yet more line-breaking and alignment when the things being pushed right are already long. Altogether that can lead to way more lines for an originally compact piece of code.
I say it's harmful to editability because alignment means you need to update a bunch of other lines when you made a change to something like the name of function that is called with aligned arguments.
For me, either rustfmt defaults need to change or I'd have to adopt a custom configuration, which seems unfortunate. I'm pro-uniform formatting across the community, but rustfmt is currently a bit crazy.

Oh, I should note as well that the original code is definitely not fully consistent, especially not function signatures. I was trying out a few different styles at different times, and now I think I lean towards a style I got from @retep998.
For signatures that fit on line, nothing is done. For signatures that are too long, it becomes:
    fn assign_to_aggregate(
        &mut self,
        dest: Pointer,
        dest_repr: &Repr,
        variant: usize,
        discr: Option<u64>,
        operands: &[mir::Operand<'tcx>],
    ) -> EvalResult<()> {
        // ...
    }

I left some comments as well, trying to link some rustfmt issues. In general, I don't want to add a rustfmt.toml because I want to have One True Rust Style.
I agree that 4-space indentation is better than visual alignment—although I have to admit that I like reading it (on a widescreen display). I tend to write chained method calls like this:
let long_variable_name: Result<Vec<String>, Box<Error>> =
    thingy // could also be directly after the =
    .iter().enumerate() // 'simple' method calls in one line
    .map(|(i, x)| {
        /* do stuff */
        /* possibly multiple lines, themselves indented */
    ).collect(); // no need to put this on a different line, also visually end the .map

(Well, I should've probably written that in a rustfmt issue instead.)

I left some comments as well, trying to link some rustfmt issues. In general, I don't want to add a rustfmt.toml because I want to have One True Rust Style.

Thanks for working to improve rustfmt! I would also like to have one community style, and I'm generally willing to accept things I don't prefer to get there, within reason (including some, but not all of the things I complained about on this PR).

I agree that 4-space indentation is better than visual alignment—although I have to admit that I like reading it (on a widescreen display).

I use widescreen displays, but I use them so I can have multiple files side by side, or an editor and a terminal. So I have a relatively narrow effective display, generally enough for 100 chars wide in a given buffer, and rustfmt's alignment really pushes a lot of code to that breaking point.

I tend to write chained method calls like this:

This is basically exactly how I like them. These indentation rules seem simpler to reason about and implement to me, too. Especially considering that every editor needs to implement something approximating rustfmt style if we want programmers to actually use the style (it makes no sense to write in one style and then convert - you want your editor to handle the code as it should be).

I'll close this PR, it was only meant to gather some feedback for rustfmt.
When some changes to rustfmt are release (e.g. https://github.com/rust-lang-nursery/rustfmt/pull/960), we can try again 😄

Sounds good.


PropertyFieldViewPicker - Unable to load Views from another site

Category

[ ] Enhancement
[X ] Bug
[ ] Question

Version
Please specify what version of the library you are using: [    1.18.0     ]
Expected / Desired Behavior / Question
I have a web part with Site Url, PropertyFieldListPicker and PropertyFieldViewPicker. When a Site Url is set, the List Picker displays the lists from that site (using the webAbsoluteUrl property).
Expected this also to be the case with the View Picker.
Observed Behavior
Even if webAbsoluteUrl is set (to another site other than the current site), it does not load the views for the selected list in the List Picker.

Error message from console:
| (anonymous) | @ | PropertyFieldViewPickerHost.js:62
-- | -- | -- | --
  | Promise.then (async) |   |
  | sch4.PropertyFieldViewPickerHost.loadViews | @ | PropertyFieldViewPickerHost.js:60
  | sch4.PropertyFieldViewPickerHost.componentDidMount | @ | PropertyFieldViewPickerHost.js:45
  | Na | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | Ma | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | (anonymous) | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | t.unstable_runWithPriority | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | To | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | xo | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | bo | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | mo | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | Wa | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | Co | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | Po | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | Do.render | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | (anonymous) | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | Eo | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | Uo | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | render | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | (anonymous) | @ | sp-property-pane_en-…7257c0677c6da0.js:1
  | Promise.then (async) |   |
  | e._loadPropertyPaneComponent | @ | sp-property-pane_en-…7257c0677c6da0.js:1
  | e._buildAndLoadPropertyPane | @ | sp-property-pane_en-…7257c0677c6da0.js:1
  | (anonymous) | @ | sp-property-pane_en-…7257c0677c6da0.js:1
  | Promise.then (async) |   |
  | e._renderPropertyPane | @ | sp-property-pane_en-…7257c0677c6da0.js:1
  | e._executeAction | @ | sp-property-pane_en-…7257c0677c6da0.js:1
  | e.requestAction | @ | sp-property-pane_en-…7257c0677c6da0.js:1
  | e.requestPropertyPaneAction | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | n.handleConfigureButtonClicked | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | handleConfigureButtonClicked | @ | chunk.sp-canvas-edit…01ad640fa1e1b2.js:1
  | onClick | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | u | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | m | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | (anonymous) | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | S | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | O | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | P | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | D | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | En | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | _o | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | Le | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | On | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | (anonymous) | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | t.unstable_runWithPriority | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | So | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
  | Nn | @ | sp-webpart-workbench…d8c6b3cd80d4e2.js:1
Steps to Reproduce
Add the following controls to a web part:
Site Url - Text Field
List - PropertyFieldListPicker
View - PropertyFieldViewPicker
When Site Url is set to another site than the current site, views fail to load based on the selected List.
Submission Guidelines

Hi, no feedback on this issue yet.
@hugoabernier, do you have any feedback on this?

Hi again,
I downloaded the source code, and added the ViewPicker to my project.
After debugging, I tested with adding the following:
webAbsoluteUrl: properties.webAbsoluteUrl,

I added it (on line 135) in the export function PropertyFieldViewPicker in PropertyFieldviewPicker.ts.
This seems to do the trick.

@frankok thanks for reporting this. Sorry, with the massive numbers of emails I get from GitHub, this one slipped through the cracks.
Working on a PR to include your fix. Thank you!

Hi @frankok!
Thank you for reporting this one!
@hugoabernier made the fix and it'll be included in the next release.
In a meanwhile you can test it in beta version.


Typos in various documentation and script files

Report
Typos in various documentation and script files need fixing.
Expected behavior
No response
Steps to reproduce the problem

CONTRIBUTING.md

Correct "We opened sourced Pezzo" to "We open sourced Pezzo"

libs/client/README.md

Correct "licesnse" to "license"

README.md

Correct "simplay" to "simply"

libs/common/scripts/sync-version.ts

Correct "Syncing version" to "Synchronizing version"

Logs (if applicable)
No response
Pezzo version
0.8.0
How do you use Pezzo?
Local Development Setup

Hey @arielweinberger, i have raised the PR to address the issue. Please have a look.


Obsolete and no longer available UX for application insights

Event with respect, that classic metrics retired- this article demonstrated out-of-dated experience of Application Insights that could be actualized.
Please, take a look on all images.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 90f43045-6caa-3b9b-f293-85e28c71372e
Version Independent ID: 47ea0b3f-96c7-7f36-07f7-f2ea3b077330
Content: Exploring Metrics in Azure Application Insights - Azure Monitor
Content Source: articles/azure-monitor/app/metrics-explorer.md
Service: azure-monitor
Sub-service: application-insights
GitHub Login: @mrbullwinkle
Microsoft Alias: mbullwin

@SychevIgor Thanks for the feedback.
Assigning content owner @mrbullwinkle for visibility, review and document enhancement consideration as appropriate.

Thank you for reporting this. This article has been retired and all references have been redirected to the appropriate up-to-date articles.

#please-close


Bump rendering dep to 8

🎉 New feature
Summary
Bump main  to depend on rendering8
https://github.com/gazebo-tooling/release-tools/issues/853
Related PRs:

https://github.com/gazebo-tooling/gazebodistro/pull/124
https://github.com/osrf/homebrew-simulation/pull/2125

Checklist

[x] Signed all commits for DCO
[ ] Added tests
[ ] Added example and/or tutorial
[ ] Updated documentation (as needed)
[ ] Updated migration guide (as needed)
[ ] Consider updating Python bindings (if the library has them)
[ ] codecheck passed (See contributing)
[ ] All tests passed (See test coverage)
[ ] While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

@osrf-jenkins run tests please

This PR is ready. Ubuntu Focal / Jammy, Homebrew, Windows CI builds should all be working


drt/ta: Remove duplicate single threaded and multi threaded code

This PR seems to be based on a misunderstanding of the batching.  The batch is not about limiting parallelism (which openmp does) but about limiting memory usage.  We write the results of a batch back to the db and then release the batch before starting a new one.   The workers consume more memory while they are working than the final results they produce.
dr has the same strategy.

This PR seems to be based on a misunderstanding of the batching. The batch is not about limiting parallelism (which openmp does) but about limiting memory usage. We write the results of a batch back to the db and then release the batch before starting a new one. The workers consume more memory while they are working than the final results they produce.

Makes sense, I missed that aspect. Most of the PR is removing the duplication of the MAX_THREADS=1 and MAX_THREADS!=1 cases, is it ok to resubmit with the batching restored?

The current setup will be slightly more efficient for threads=1 due to avoiding locking & omp but I think simplicity is better here.  This is likely for having a reference for testing and reporting threading speedup over the base case.  Updating is fine.

I've taken out the patch that removes the batching. There is another patch in the updated series that removes some unused code that hasn't been reviewed yet.


Telepathy + Silk Touch

So this one is pretty big,
If a player has Silk Touch & Telepathy on their pick they are able to mine spawners regardless of if they have the permissions to do so or not.
Ie - A ranked player is meant to be the only player that can Mine Spawners with Silk Touch.
But a normal player and use the Silk Touch + Telepathy combo and mine the spawners.

Alright thanks ill look into it.

Thanks <3

Fixed and will be in the next update.


making scans faster

Just putting this idea here not to forget:
For scans (e.g. 1-d nutau norm, 2-d contours, ...) we always start the minimizer from nominal, but it would converge much faster if started from a neighboring best-fit point.
Example: when we scan nutau norm from 0 to 2 in steps of 0.1, then we can do a first fit at 0 starting from nominal settings, and then use this bestfit value to seed the next fit at 0.1, and so forth.....could save quite some time

Closing, since we've been doing ok with the current setup..


Large Material Press does not manufacture double plates

I have set the MetalBender mode and as I understand it, it does not require any chips at all to select a recipe. I play the Nomifactory GTCeu assembly and when I tried to create a double plate (any one), he gave me a regular plate if there is an ingot, and a foil if there is a plate. I chose the right chip for the manufacture of dual plates, but apparently it uses only the mode of ordinary plates.

I have set the MetalBender mode and as I understand it, it does not require any chips at all to select a recipe. I play the Nomifactory GTCeu assembly and when I tried to create a double plate (any one), he gave me a regular plate if there is an ingot, and a foil if there is a plate. I chose the right chip for the manufacture of dual plates, but apparently it uses only the mode of ordinary plates.

Oh, I removed all the chips from the additional ports with other configurations and left only the one where the dual plates are needed and it worked. I just thought that, depending on the port, he would use those chips that lie in that port, and not just one.

This problem is also observed with the extruder

This problem is also observed with the extruder

I think that if I start checking all the other machines, then I think they will also mix recipes, although there is no such thing with an assembly machine, thank God.

This problem is also observed with the extruder

I think that if I start checking all the other machines, then I think they will also mix recipes, although there is no such thing with an assembly machine, thank God.


Add new IAM user(Ryan Sakuma)

I have followed the pre-work checklist issue template for new Devops members. I have successfully initialized terrraform locally with the architecture specified, and am adding myself to the AWS user files. Please comment or provide any feedback if there is anything that I can do to improve this PR.

Admin: run the following command from the AWS CLI:
aws iam create-login-profile --user-name USERNAME --password PASSWORD --password-reset-required


Waiting for .plug.kak.lock

I regularly have […]: Waiting for .plug.kak.lock next to my topmost plugin in the plugin list (e.g. when I plug-update.
What does this mean and do I need to do something about it?
I always get the feeling that the topmost plugin then needs to mbe removed and reinstalled to update.

Sorry for the wait. I'm not the maintainer of this plugin, but I do have some ideas.
If I recall correctly, the reason that the .plug.kak.lock file is used is to inform plug.kak that an operation is underway, and this file is removed once the operation is finished. This allows for asynchronous operation of various commands in the background while still preventing execution of operations while another one is still underway.
I think the problem you're having is a similar one to one I had for a while, whixh is that you may have quit Kakoune while plug.kak was performing an update or some other operation, and this file didn't get cleared. Try navigating to your ~/.config/kak/plugins directory (or wherever your plugins are located) and running rm -f .plug.kak.lock, and then restarting Kakoune to see if that fixes things.

Sorry for the wait. I'm not the maintainer of this plugin, but I do have some ideas.
If I recall correctly, the reason that the .plug.kak.lock file is used is to inform plug.kak that an operation is underway, and this file is removed once the operation is finished. This allows for asynchronous operation of various commands in the background while still preventing execution of operations while another one is still underway.
I think the problem you're having is a similar one to one I had for a while, whixh is that you may have quit Kakoune while plug.kak was performing an update or some other operation, and this file didn't get cleared. Try navigating to your ~/.config/kak/plugins directory (or wherever your plugins are located) and running rm -f .plug.kak.lock, and then restarting Kakoune to see if that fixes things.

I think that solved it, thanks!

I think that solved it, thanks!


[Type checker] Don't overflow array bounds with #fileLiteral

Resolves rdar://problem/26586984.

@swift-ci please smoke test and merge


Added status log when closing WebRTCConnections

We should now always have an status: 105 message when connections are closed.
Regarding the implementation, I'm beginning to think we should wrap the addon.WebRTCConnection object in JS and explicitly add the context to it. Init and close should be functions of that object. What we have now is a little awkward.

@lodoyun with this PR it will be a lot easier to track status, nice work!


feat(test): switch meta to sql backend in simluate test

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
this pr switch simulate test from etcd-backend meta to sql-backend meta with a in-memory sqlite.
this pr should update Cargo.toml after merging of https://github.com/madsim-rs/sqlx/pull/1 and https://github.com/madsim-rs/madsim/pull/197

Checklist

[ ] I have written necessary rustdoc comments
[ ] I have added necessary unit tests and integration tests
[ ] I have added test labels as necessary. See details.
[ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
[ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
[ ] All checks passed in ./risedev check (or alias, ./risedev c)
[ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

[ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

[ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Found in madsim integration test:
thread '<unnamed>' panicked at src/meta/src/controller/streaming_job.rs:1245:21:
internal error: entered unreachable code: sql backend doesn't support custom parallelism

cc @shanicky do we plan to support it soon?

Two cases failed in end-to-end test (parallel).
Cc @zwang28 could you please help to take a look?
e2e_test/streaming/grouping_sets/grouping_sets.slt           .. [FAILED]
  |
  | failed to run `e2e_test/streaming/grouping_sets/grouping_sets.slt`
  |
  | Caused by:
  | query failed: db error: ERROR: Failed to run the query
  |
  | Caused by these errors (recent errors listed first):
  | 1: gRPC request to batch service failed: Internal error
  | 2: Storage error
  | 3: Hummock error
  | 4: Expired Epoch: watermark<PHONE_NUMBER>508544, epoch<PHONE_NUMBER>180864.
  |
  | [SQL] select * from v;
  | at e2e_test/streaming/grouping_sets/grouping_sets.slt:13
  |
  |
  | e2e_test/streaming/dynamic_filter.slt                        .. [FAILED]
  |
  | failed to run `e2e_test/streaming/dynamic_filter.slt`
  |
  | Caused by:
  | query failed: db error: ERROR: Failed to run the query
  |
  | Caused by:
  | Storage error: Hummock error: Expired Epoch: watermark<PHONE_NUMBER>508544, epoch<PHONE_NUMBER>180864.
  |
  | [SQL] select * from mv1 order by v1;
  | at e2e_test/streaming/dynamic_filter.slt:75
 ```

Two cases failed in end-to-end test (parallel).
Cc @zwang28 could you please help to take a look?

🥵 I cannot reproduce it yet.
Maybe should handle the deterministic e2e failure first and hopefully it subsequently can reproduce this one.

Any updates? 🥵

@KveinAxel said the failure of recovery test was still not resolved. 🥵

Dependency issues have been fixed. Now looking into failing tests. https://buildkite.com/risingwavelabs/pull-request/builds/55574#01914bb5-645a-49f4-87cd-c1b1ffe99625

What I'm going to do is to get this tests merged first. Because now it already works. But it's blocked by bugs instead.
Then we can make incremental progress.

I'm not gonna fix the bugs in this PR. Instead just try and get the test merged first.
Subsequently, we can add the labels:

ci/main-cron/run-selected
ci/run-recovery-test-deterministic-simulation

To run the tests in a PR.
@BugenZhao could you PTAL to see if the dependencies are patched correctly? I'm waiting on https://github.com/madsim-rs/sqlx/pull/2 to get merged. In the meantime I use a forked dependency for sqlx.
@yezizp2012 could you PTAL at the simulation tests part? I preserve the etcd configurations, until we deprecate etcd. We shouldn't remove these tests totally.
Finally, we use an on-disk sqlite. I don't think this is totally right. But it's good enough to get the recovery tests working.
Later we can improve it to use a shared in-memory cache sqlite. But it will need to be spawned separately.

Or @xxchan if you're free, PTAL at the patched deps for cargo lock.


Invalid tenant id. Tenant have been mutated in lowercase, and this is bad.

Hi @mvertopoulos ,
the tenandId have been converted in a lowercase string. The assignment is correct.
The configuration:
Vue.use(msal, {   auth: {     clientId: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',     tenantId: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',     redirectUri: 'localhost:8080',     validateAuthority: false,   } });
Request URL: https://login.microsoftonline.com/xxxxxxxxxxxxxxxxxxxxxxxx/v2.0/.well-known/openid-configuration 400 Bad Request
{    "error":"invalid_tenant",    "error_description":"AADSTS90002: Tenant 'xxxxxxxxxxxxx' not found. This may happen if there are no active subscriptions for the tenant. Check to make sure you have the correct tenant ID. Check with your subscription administrator.\r\nTrace ID: **********************************\r\nCorrelation ID: ******************************\r\nTimestamp: 2020-05-08 17:54:51Z",    "error_codes":[       90002    ],    "timestamp":"2020-05-08 17:54:51Z",    "trace_id":"**********************************",    "correlation_id":"***********************************",    "error_uri":"https://login.microsoftonline.com/error?code=90002" }

Duplicate of #11


Component lifecycles differs on nativescript-angular and angular2 on web.

I ran into this problem and I'll try to explain it here.
With nativescript-angular the lifecycle hook ngOnDestroy isn't called when we navigate away from a page as it would be on web. Navigating back reuses the old component with nativescript-angular but a new instance is created on web.
I've based an example on Nathan Walker's angular2-seed-advanced:
https://github.com/m-abs/angular2-seed-advanced/tree/ngondestroy-not-triggered
I've two component for my routes:
HomeComponent
AboutComponent
Each component have a instanceNo, so we can identify multiple instances of the component.

Action
Web
TNS

Launch application
HomeComponent<1>.ngOnInit() is called
HomeComponent<1>.ngOnInit() is called

Click on 'ABOUT'
HomeComponent<1>.ngOnDestroy() followed by AboutComponent<1>.ngOnInit()
AboutComponent<1>.ngOnInit()

Click on 'HOME'
AboutComponent<1>.ngOnDestroy() followed by HomeComponent<2>.ngOnInit()
HomeComponent<2>.ngOnInit()

Click back
HomeComponent<2>.ngOnDestroy() followed by AboutComponent<2>.ngOnInit()
HomeComponent<2>.ngOnDestroy()

Click back again
AboutComponent<2>.ngOnDestroy() followed by HomeComponent<3>.ngOnInit()
AboutComponent<1>.ngOnDestroy()

As you can see the two behaves very differently.
On web:

Navigate to a new page, the current component is destroyed and a new is created.
Navigating hitting back, the current component is destroyed and a new is created.

With nativescript-angular:

Navigate to a new page, creates the new component but leaves the current component as is.
Navigating hitting back, the current component is destroyed and the old component is reused.

If you want to try for yourself, checkout my branch ngondestroy-not-triggered and run:
For web:
npm run start

For nativescript-angular with android:
npm run start.livesync.android

Shouldn't the two behave the same?
So components are destroyed when we navigate from a page and (re)created when we navigate to the page?

@m-abs You are indeed correct. When doing a page navigation in native app the page/component you are navigation from is not actually destroyed. When you navigate back to it the component is not created again. The original component is shown instead.
Can you please elaborate on you scenario and why this is a problem?
There are also a couple of other ways to plug into the app navigation:

Subscribe to the Router.events observable
Use route guards

@vakrilov
Thank you for the suggest, I'll try it when I get back to work :)
I'll can try to elaborate, sorry for the long text :)
Our project is based on angular2-seed-advanced, so we share code between the {N} -version and the web-version of our project.
The project is an audiobook-player that shows HTML-content synced to the audio during playback.
We render the HTML-content with our content-viewer-component. On web we use the component as is and in {N} it runs in a small ng2-webapp inside a web-view.
Our page-component sets up events and subscriptions to make this work in our ngOnInit() hook and they’re removed in our ngOnDestroy()hook.
Since the page-component lives on after navigation from it, the subscriptions and events created in ngOnInit() will still be active, this means that the web-view/content-viewer-component still does work that is now unneeded.
We also risk having two instances of the web-view/content-viewer-component running at the same time. If the user navigates to the page again without going back, since that won’t reuse the component.
I also think the behavior goes against what the ng2-docs imply.
From: https://angular.io/docs/ts/latest/guide/lifecycle-hooks.html#!#hooks-overview

hook
Purpose

ngOnInit()
Initialize the directive/component after Angular initializes the data-bound input properties.

ngOnDestroy()
Cleanup just before Angular destroys the directive/component. Unsubscribe observables and detach event handlers to avoid memory leaks.

From: https://angular.io/docs/ts/latest/guide/router.html#!#route-parameters

Observable params and component re-use
In this example, we subscribe to the route params Observable. That implies that the route params can change during the lifetime of this component.
They might. By default, the router reuses a component instance when it re-navigates to the same component type without visiting a different component first.  The parameters can change between each re-use.

We can of cause use the Router.events observable as a workaround, but I’d much rather just rely on the hook ng2 components have for it.
I’d also prefer the same behavior (where possible) on both web and in {N}, otherwise it can be surprising for the developers.
I can see it’s done intentionally here: https://github.com/NativeScript/nativescript-angular/blob/master/nativescript-angular/router/page-router-outlet.ts#L112
And I do understand the benefits of caching components like that, but what about larger app with many pages where the user don’t necessarily navigates back?
Don't we risk it accumulates old components and thus causing the leaks the ng2-docs mention?

+1

Maybe some intermediate approach is when using the paged navigation indicate which is the expected reusability of the components (I would say destroy as default and preserve as alternative).

Any updates on this ? Very urgent!

We are using it in a soon to be launched app and we have already invested too much time with nativescript to go back now. This issue is going to be a dealbreaker. I am willing to help if someone can let me know what do do.

Our designer wants a three tabbed interface, where the navbar is always visible at the bottom of the screen and the user can tap on each tab.
So the user can always forward navigate to any of our three tabs.
The way the router caches pages at the moment and since child-routes doesn't work right on back (see #477) we'll will end up with memory leaks it we don't find a solution.
I've been considering using an JS-hack to work around this by making the components constructor return the first component, but this is not a solution I like.
@Component({...})
export class PageComponent {
  private static singleton: PageComponent;
  constructor(...) {
    if (PageComponent.singleton) {
      return PageComponent.singleton;
    }

     PageComponent.singleton = this;
  }
}

There're a bunch of problems with this solution, like potentially breaking DI, since the first components injections will always be used. and ngOnInit would get called multiple times.
I've also considered making a private fork of the ns-router that doesn't have caching feature, but I really don't want to maintain it.

Any way to temporarily disable cache folks ? This issue is really the breaking point and we need to launch our app ASAP. Can someone please help out. Thanks in advance

@sivsushruth
If you use a router-outlet and managed back-buttons yourself, pages aren't cached. (Since it's not a NativeScript Page switch).

@sivsushruth
Unfortunately you need to capture back-button on android (like this: https://github.com/m-abs/angular2-seed-advanced/blob/child-router-android-back-button/nativescript/app/pages/app/app.component.ts#L58-L69) and add your own NavigationButton to the ActionBar.

@m-abs The issue however is, I need caching for a few pages, is it possible to have router-outlet inside page-router-outlet ?

@sivsushruth
That's what we do.
We've created a few child routes like this: https://github.com/m-abs/angular2-seed-advanced/blob/child-router-android-back-button/src/client/app/components/home/home.routes.ts
And since we're using child-routes, we needed to add an router-outlet to our HomeComponent: https://github.com/m-abs/angular2-seed-advanced/blob/child-router-android-back-button/src/client/app/components/home/home.component.tns.html#L23
Routes under /home/ in our example are not cached.

+1 for ng2 lifecycle or additional hooks to determine cache state of page. This would be helpful to subscribe / desubscribe from observables (e.g. if you navigate by observable return on pages, you get hit by race conditions to which page you actually will be navigated).

@tviel You can listen to router.events, you can't use ActivatedRoute, because it seems to stop emitting events after first navigate away from the component...
Could we please make this behavior optional? It makes sense to cache some pages but not others.

@tviel Update on my last comment.
Router.events didn't work for us, but I can listen to Page.unloadedEvent and Page.loadedEvent to see if a component is unloaded.

@m-abs I was also able to workaround this using :
        this.page.on(Page.unloadedEvent, event => {
            this.ngOnDestroy();
        })

But it definitely seems like overhead.

Let's do a little summary so far
Subscriptions And CleanUp
When using page-navigation ngOnInit() will be called the first time the component is navigatedTo and ngOnDestroy() will be called when navigating back from the page with the component (when it is removed form the navigation stack). When navigating forward(deeper in the app) those hooks will not be called. The component instance will be preserved as it is still connected to a page that lives inside the native navigation stack.
However, you might want prevent it from updating while not shown. So far I think @m-abs solution of using page events would be the simplest. You might call the ngOnInit()/ngOnDestroy() hooks directly, but it's good idea to guard them with a flag to prevent double execution.
Using ActivatedRoute and Params
Normally, in Angular you would inject ActivatedRoute and read parameters from it. Your component will be reused if you do a subsequent navigations to the same route while only changing the params. That's why params and data inside ActivatedRoute are observables.
In NativeScript when navigating back to an existing page your component will not be re-created. Angular router will still create an new instance ActivatedRoute and put all params in it, but you cannot get hold of it trough injection as no constructor is called.
The Solution: In NativeScript you can inject PageRoute which has an activatedRoute: Observable<ActivatedRoute> inside it. Each time a new ActivatedRoute instance is created for this re-used component it will be pushed in this observable, so you can still get you params:
So, instead of:
class MyComponent {
  id: number;
  constructor(private route: ActivatedRoute) {
    this.route.params
      .forEach((params) => { this.id = +params['id']; });
    }
}

You do:
class MyComponent {
  id: number;
  constructor(private pageRoute: PageRoute) {
    // use switchMap to get the latest activatedRoute instance
    this.pageRoute.activatedRoute
      .switchMap(activatedRoute => activatedRoute.params)
      .forEach((params) => { this.id = +params['id']; });
  }
}

Why So Complicated
Native platforms preserve the views in the native navigation stack when navigating deeper. So we need to preserve the components that are connected to those views too. This is done in the <page-router-outlet>. However, there are some limitations in terms of how Nativescript plugs into the router live-cycle explained in angular/angular#7757. I hope in future version of angular these will be resolved so we can narrow the gap between using <page-router-outlet> and the stock <router-outlet>

@m-abs
Thanks :D
So great, use loadedEvent, unloadedEvenet in page instance instead ngOnInit and ngOnDestroy solve all problems

Awesome! Page.unloadedEvent on ngOnInit  solved my problem.  @m-abs Thanks alot.

It still doesn't work for me. Any idea where I'm doing wrong?
import { Component, OnDestroy } from "@angular/core";
import { ActivatedRoute } from "@angular/router";
import { ListingService } from "../../shared/listing/listing.service";
import { Listing } from "../../shared/listing/listing";
import { RouterExtensions, PageRoute } from "nativescript-angular/router";
import "rxjs/add/operator/switchMap";
import {Page} from "ui/page";

@Component({
  selector: "listing-detail",
  providers: [ListingService],
  templateUrl: "components/listingDetail/listingDetail.html",
  styleUrls: [
    "components/listingDetail/listingDetail-common.css",
    "components/listingDetail/listingDetail.css"
  ]
})
export class ListingDetailComponent {
  public id: number;
  public listing: Listing;
  constructor(
    private pageRoute: PageRoute,
    private page: Page,
    private listingService: ListingService,
    private routerExtensions: RouterExtensions
  ) {
    // Get listing ID
    this.pageRoute.activatedRoute
      .switchMap(activatedRoute => activatedRoute.params)
      .forEach((params) => { this.id = +params['id']; });
  }

  ngOnInit() {
    this.listingService.get(this.id).subscribe(loadedListing => {
      this.listing = loadedListing;
    });

    this.page.on(Page.unloadedEvent, event => {
        this.ngOnDestroy();
    })
  }

  public onBackButtonTap() {
    this.routerExtensions.back();
  }

  ngOnDestroy() {

  }
}

@bunnyvishal6 @m-abs able to help?

@MrNicodemuz
I think the problem is you don't call ngOnInit() after you change the id. So you don't load the new data.
Sent from my OnePlus ONEPLUS A3003 using FastHub

@m-abs many thanks it's working now! :)

@m-abs @gergderkson Guys, my app is closed after call this.ngOnDestroy();
Any help is really appreciated

Hi, I solved the issue, triggering change detection in my root component, every time any navigation ends.
this.router.events.subscribe(e => e instanceof NavigationEnd && this.changeDetectionRef.detectChanges());

I wrote an article about this. I solved it in generic way with the decorator.

This remains a major limitation / "feature" :/

Why if Im writing a wizard, i need to move forward and backward, and sometimes move, skip, or jump to pages/views and limit user capabilities of moving back to pages (i could set a new root here, but not my case) not catched or in any state. Also if i complete the wizard i should be able to ngDestroy all the previous views/pages to do a clean new wizard setup. Current logic is not allowing me to do that, and i do not wanna recreate the wheel or change the native behavior, is there a way to trigger those events or angular cycle hooks (init, destroy, etc) using nativescript?


fix compilation error with literal newline (#5)

Literal newlines need to be inside backticks not double quotes.
Fixes #5.

see my comment on issue #5.  this is not quite the right fix.  (nonetheless, the real mystery is why no fix at all has been added to the master branch in the last 23 months.  there seem to be two people with commit access to this repo, but a showstopper bug hasn't been fixed.  has active development moved to some other fork?)

Good catch; updated the PR accordingly.
I guess the maintainers probably lost interest and/or don't use this tool any more.

I've also merged into the master branch of my fork in case anyone wants a working version:
https://github.com/aspiers/chrome-session-dump/tree/master


Exclude unneeded files from the C repo when packaging

I happened to be watching the output of a fresh fetch and noticed that zstd-sys was the largest crate that was downloaded at 1.9MiB and suspected it was due to including the entirety of the C lib's repo, which was indeed the case. This just excludes everything from that repo that isn't needed for actually compiling the lib, and the the LICENSE/COPYING file of course.
This makes the crate go from 1.9MiB to 632KiB, so a little less than 1/3 of the current size.

Indeed - thanks for the work!

Cool, thanks for the quick merge!


Use v3 data bag encryption

Closes: #5819
cc @chef/maintainers

might have actually broken a chef-zero test.

FWIW, the original problem was lack of availability of the encryption modes we wanted to use in older versions of OpenSSL. There was some question about non-omnibus installs technically being supported that was holding us back. Should be a-ok now though.
👍
Would be great to fix the docs also, as mentioned in the comment on original ticket: https://github.com/chef/chef/issues/5819#issuecomment-280393780